Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.020419799s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:29:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.020450428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.......X........X.............................X..........] duration: 6.71756084s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:29:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.717581449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.000997101s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:28:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.001035154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.......X........X.............................X.........] duration: 6.460918883s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:28:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.461023241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.774696999s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:27:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.77477672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.......X........X.............................X........] duration: 5.897908244s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:27:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.897940587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.861158779s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:27:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.861185289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.......X........X.............................X.......] duration: 7.129509969s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:26:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 7.129538073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.038393923s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:26:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.038420493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.......X........X.............................X......] duration: 5.985271175s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:25:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.985292275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.137181894s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:25:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.137212331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.192489209s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:25:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.192530227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.......X........X.............................X.....] duration: 6.718622665s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:25:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.718659435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.......X........X.............................X....] duration: 7.477913306s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:24:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 7.477943533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.903362734s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:24:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.903388663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.207580722s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:23:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.207614315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.......X........X.............................X...] duration: 6.262619286s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:23:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.262650407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.866082531s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:22:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.866115464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.......X........X.............................X..] duration: 6.196752688s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:22:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.196792002s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.......X........X.............................X.] duration: 8.942769582s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:21:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 8.942802282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 13.183057111s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:21:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.18308245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.......X........X.............................X] duration: 30.277088616s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:21:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.277116819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.861112314s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:20:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.861146538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......X........X.............................] duration: 6.274001641s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:19:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.274030285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......X........X............................] duration: 7.034115168s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:19:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.034147679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.093338109s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:19:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.093367525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......X........X...........................] duration: 6.504475004s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:17:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.504501233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.046921613s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:17:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.046949084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.......X........X..........................] duration: 6.157715057s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:17:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.157739993s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......X........X.........................] duration: 6.720218662s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:17:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.720353219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.505398079s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.505428466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......X........X........................] duration: 12.19396842s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:16:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 12.194000522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.394719905s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:16:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.394757777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......X........X.......................] duration: 6.78392097s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:15:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.783958142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.59811847s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:15:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.598156884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......X........X......................] duration: 6.735373042s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:14:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.735404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.08497373s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:14:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.085004318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......X........X.....................] duration: 6.394451874s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:13:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.394483684s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.273682568s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:13:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.273713605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.983520658s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:12:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.983547089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......X........X....................] duration: 6.300395537s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:12:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.300529061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.933842877s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:12:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.933868556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X........X...................] duration: 6.038726556s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:11:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.038746807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.031926559s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:11:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.032167249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 13.909516642s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:10:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.909538485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......X........X..................] duration: 10.232308088s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:10:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 10.232341671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.269857764s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:10:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.269884555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......X........X.................] duration: 6.774837434s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:09:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.774873102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X........X................] duration: 7.110126789s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:08:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.110162668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X........X...............] duration: 6.850736106s trace: https://jaeger.dev.storj.io/trace/4c440d21de358446 FAIL (2025-11-13 16:07:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 03h51m51.494948691s) [upload=91430]: Starting upload Nov 13 15:55:52.943558302 (103h51m51.495028783s) [upload=91430]: Done starting upload Nov 13 15:55:52.943886777 (103h51m51.495357238s) [upload=91430]: Uploading segments... Nov 13 15:55:52.943944347 (103h51m51.495414798s) [upload=91430]: Got next segment. Inline: false Nov 13 15:55:52.943998921 (103h51m51.495469382s) [upload=91430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 15:55:52.944003630 (103h51m51.495474091s) [upload=91430]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 15:55:53.191769449 (103h51m51.74323992s) [seg_pos={0 0}, upload=91430]: Begin upload segment... Nov 13 15:55:53.191832479 (103h51m51.74330293s) [seg_pos={0 0}, upload=91430]: Done begin upload segment. Nov 13 15:55:53.191868297 (103h51m51.743338758s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.194608144 (103h51m51.746078595s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195256228 (103h51m51.746726679s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.195805985 (103h51m51.747276446s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196040071 (103h51m51.747510532s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196165319 (103h51m51.74763578s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.196700398 (103h51m51.748170859s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197116471 (103h51m51.748586932s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.197647031 (103h51m51.749117482s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198105774 (103h51m51.749576235s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.198237485 (103h51m51.749707946s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.199646839 (103h51m51.7511173s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.200729320 (103h51m51.752199781s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201233992 (103h51m51.752704443s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.201699678 (103h51m51.753170129s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:53.202143453 (103h51m51.753613904s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Uploading piece... Nov 13 15:55:54.982460026 (103h51m53.533930477s) [upload=91430]: Next returned nil segment Nov 13 15:55:54.982477849 (103h51m53.53394831s) [upload=91430]: Waiting for error group managing segments... Nov 13 15:55:55.296513130 (103h51m53.847983581s) [noise=true , address=77.90.31.10:4520 , piece_id=UCC7NQOGM54UGO3KNHSRJHQMR37BPXYLJNOW7MDATZYTTT73H75A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.348160427 (103h51m53.899630888s) [noise=true , address=45.91.116.8:4047 , piece_id=YCZZAO225XNQBX3KLXYQECYLKUEIV4AEEAZS2EZINATKNA765LKA, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.381440720 (103h51m53.932911181s) [noise=true , address=77.90.31.10:4517 , piece_id=PT4NO7VDJ7I4AWNJYJLU4ROE4UDCXTGJEK4GTIVR4A3EASARNZ7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.404680153 (103h51m53.956150614s) [noise=true , address=77.90.31.10:4518 , piece_id=6XCVBKMWSOCOKQAMQXPJM3KYDBER6IWBV22IEO523PZRZA2FYZ4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:55:55.710847831 (103h51m54.262318293s) [noise=true , address=45.91.116.12:4225 , piece_id=RCLSR3KKMWUTALSW6F6H7LQ5RNFEFTCMZ5JFAQ2DUN6TVA3U7B5A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:13.913136443 (103h52m12.464606904s) [noise=true , address=145.79.227.19:4078 , piece_id=OUFE7BHY6BSEDLXKA7TYOJ5BTSCLAC4GJDPJSHLBT3IXXF3O5XHQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:15.504274489 (103h52m14.05574494s) [noise=true , address=145.79.227.27:4690 , piece_id=NSE4TC6CYIYRQZSOR5MOTVTKG5EHRAPQE3YTB6QO6CLYY5V4NZRQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.460240854 (103h52m15.011711305s) [noise=true , address=145.79.227.16:4910 , piece_id=6K7OPFNZUET4XKYGHXJEIDKE532DP7GPUYHM37AJD265K3FN6AVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:16.925967647 (103h52m15.477438108s) [noise=true , address=145.79.227.32:4834 , piece_id=KTQAQVUW3YDOWPB3GBMDR3EHOSB32UL4XKA5ZPQTPZ5RUT4LEP4Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:19.210746785 (103h52m17.762217246s) [noise=true , address=145.79.227.20:4437 , piece_id=NJXKEBMD4Z3SF4PNAFOEDBDO6HH5VLINXMEJLVLK4YHPJ4HZRC2A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: <nil> Nov 13 15:56:22.253904255 (103h52m20.805374726s) [noise=true , address=145.79.227.21:4475 , piece_id=HZR7ZNDTPLXMXXGKZ6DRVLTXS2TP4O7H5OERCBTSSK7U5JQTBUVQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 15:56:22.254354673 (103h52m20.805825134s) [noise=true , address=145.79.227.30:4763 , piece_id=6LP4E5HOMHFOJECWXNGGGOJIGZNQCEB25ARCGRSZZRKT6QDGTC7A, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->145.79.227.30:4763: use of closed network connection Nov 13 15:56:22.254522512 (103h52m20.805992973s) [noise=true , address=145.79.227.10:4401 , piece_id=QACX74ZZZDXVUA4UWCIM6BRMVSNQ26J62CIEQAOBPFSRGIDW6K5Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 15:56:22.254830158 (103h52m20.806300619s) [noise=true , address=145.79.227.26:4654 , piece_id=UDCEYHPPBR6IQ35Y6VOOQFHYBQEIJWYOG4Y7QCZY3O7JG35JWZJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53368->145.79.227.26:4654: use of closed network connection Nov 13 15:56:22.254948604 (103h52m20.806419065s) [noise=true , address=145.79.227.13:4802 , piece_id=KVZFEWLBKZXABQDK65Z43MVLSCUJEYJZ5T2XUZOG3RVO4KQAXB6Q, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.13:4802: use of closed network connection Nov 13 15:56:22.255292989 (103h52m20.80676345s) [noise=true , address=145.79.227.31:4799 , piece_id=52SLFAIFUI2N7AV27WXRRMZ3BCZQAIC2EU6IMHQ2ZWKWBKH7LDJQ, seg_pos={0 0}, upload=91430]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:45736->145.79.227.31:4799: use of closed network connection; context deadline exceeded Nov 13 15:56:22.255326864 (103h52m20.806797325s) [seg_pos={0 0}, upload=91430]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 13 15:56:22.506996058 (103h52m21.058466519s) [upload=91430]: Done uploading segments... Nov 13 15:56:22.507040041 (103h52m21.058510502s) [upload=91430]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.850752407s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)