Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.380415356s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:16:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.380446305s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 309.548309ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/689205e0dee01a3b FAIL (2025-11-10 18:16:48 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39970->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 309.578146ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 9.125485665s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:16:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.12551499s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 60.316016ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:16:21 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 60.413761ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 70.419163ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:15:42 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 70.437569ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 8.330800686s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:15:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.330832477s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 307.460393ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/689205e0dee01a3b FAIL (2025-11-10 18:15:36 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39970->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 307.497774ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 68.655265ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:15:08 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.723865ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 316.355481ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/689205e0dee01a3b FAIL (2025-11-10 18:14:40 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39970->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 316.489807ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 13.660637003s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:14:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.660666879s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 315.940522ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/689205e0dee01a3b FAIL (2025-11-10 18:14:23 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39970->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 315.968685ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 55.440582ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:14:04 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 55.470227ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.420996918s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:13:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.421014673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.111113185s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:13:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.111145566s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 421.155945ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/689205e0dee01a3b FAIL (2025-11-10 18:13:39 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39970->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 421.187786ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 74.286628ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:13:29 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 74.316455ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.429635229s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:12:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.429666178s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 60.125012ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:12:40 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 60.153526ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 66.124805ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:11:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 66.154904ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.076395328s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.076424404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.318400095s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:11:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.318431224s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 61.07942ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:11:15 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 61.106822ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.737475608s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.737503813s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [X............X.] duration: 64.440198ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:10:29 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.558074ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.826901326s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:10:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.826929138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.537288138s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:09:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.537302414s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.X............X] duration: 277.077758ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ab69d2d7a727607 FAIL (2025-11-10 18:09:35 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:46774->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 277.094791ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 63.166983ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:09:09 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 63.20742ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.12651646s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:08:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.126535516s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 85.664354ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:08:39 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 85.686627ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 62.245139ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:07:58 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 62.277529ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.467077526s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:07:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.467109609s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 79.017566ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:07:22 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 79.060118ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 11.765562519s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:06:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 11.765708757s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 68.933254ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:06:31 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.9626ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 84.598715ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:06:08 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 84.637608ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 60.049347ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:05:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 60.078342ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.464629048s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:05:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.464660307s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 66.751991ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:05:18 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 66.785835ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.221750208s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:05:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.22182489s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 63.331026ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:04:57 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 63.344221ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.543685718s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:04:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.543704554s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 80.203886ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:04:31 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 80.234484ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 7.4070163s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:04:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.407048862s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 91.940786ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:04:04 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 91.974761ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.354343318s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:03:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.354375719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 11.960891498s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:02:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 11.960926807s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 63.587365ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:02:53 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 63.619505ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 689.009088ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6cf7142210ae68a0 FAIL (2025-11-10 18:02:18 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59362->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 689.038524ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.864924997s trace: https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6 FAIL (2025-11-10 18:01:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment... Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment. Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece... Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments... Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805 , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009 , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772 , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974 , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626 , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045 , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663 , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367 , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655 , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835 , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401 , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221 , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115 , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil> Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces. Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971 , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876 , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763 , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments... Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.864944685s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)