Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X...XX......X.........................] duration: 7.248000927s FAIL (2025-09-23 14:23:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.248036825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X...XX......X........................] duration: 6.970321443s FAIL (2025-09-23 14:23:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.970353936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X......X..............................] duration: 7.169057145s FAIL (2025-09-23 14:23:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 7.169091821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X...XX......X.......................] duration: 8.561836782s FAIL (2025-09-23 14:22:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.561870497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......X.............................] duration: 6.063983701s FAIL (2025-09-23 14:22:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.064019089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X...XX......X......................] duration: 14.016893601s FAIL (2025-09-23 14:22:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 14.016928737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X...XX......X.....................] duration: 7.80780261s FAIL (2025-09-23 14:21:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.807835874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X......X............................] duration: 6.092108897s FAIL (2025-09-23 14:21:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.092143493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X...XX......X....................] duration: 8.452372781s FAIL (2025-09-23 14:20:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.452424229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......X...........................] duration: 6.190707917s FAIL (2025-09-23 14:20:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.190744417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X...XX......X...................] duration: 7.137693995s FAIL (2025-09-23 14:20:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.137732107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......X..........................] duration: 6.186774778s FAIL (2025-09-23 14:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.186849219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X...XX......X..................] duration: 6.850238313s FAIL (2025-09-23 14:19:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.850271656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......X.........................] duration: 8.085832269s FAIL (2025-09-23 14:18:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 8.085867796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......X........................] duration: 7.808366365s FAIL (2025-09-23 14:18:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 7.808398867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X...XX......X.................] duration: 8.532775178s FAIL (2025-09-23 14:17:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.53281798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......X.......................] duration: 5.966734869s FAIL (2025-09-23 14:17:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.966769445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X...XX......X................] duration: 7.315781983s FAIL (2025-09-23 14:17:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.315816328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......X......................] duration: 6.024928451s FAIL (2025-09-23 14:16:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.02496463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...............................X...XX......X...............] duration: 6.522194244s FAIL (2025-09-23 14:16:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.522225343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......X.....................] duration: 5.983558658s FAIL (2025-09-23 14:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.983592913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...............................X...XX......X..............] duration: 7.179905899s FAIL (2025-09-23 14:16:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.179944022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X......X....................] duration: 5.951379699s FAIL (2025-09-23 14:15:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.951415928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...............................X...XX......X.............] duration: 6.960431016s FAIL (2025-09-23 14:15:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.960465433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......X...................] duration: 5.815045943s FAIL (2025-09-23 14:14:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.815091049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...............................X...XX......X............] duration: 6.069538121s FAIL (2025-09-23 14:13:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.069570443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......X..................] duration: 6.265190006s FAIL (2025-09-23 14:13:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.265215695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...............................X...XX......X...........] duration: 10.135365781s FAIL (2025-09-23 14:12:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.13539728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......X.................] duration: 5.963515415s FAIL (2025-09-23 14:12:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.963548769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X......X................] duration: 6.535671897s FAIL (2025-09-23 14:12:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.535952951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...............................X...XX......X..........] duration: 7.233584832s FAIL (2025-09-23 14:12:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.233619287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X......X...............] duration: 16.808179232s FAIL (2025-09-23 14:12:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 16.808217103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...............................X...XX......X.........] duration: 10.118256091s FAIL (2025-09-23 14:11:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.118288152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...............................X...XX......X........] duration: 8.721116872s FAIL (2025-09-23 14:10:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.721166768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X......X..............] duration: 7.27028746s FAIL (2025-09-23 14:10:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 7.270314411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...............................X...XX......X.......] duration: 6.611856236s FAIL (2025-09-23 14:10:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.611884771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X......X.............] duration: 9.950800382s FAIL (2025-09-23 14:10:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 9.950831963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X......X............] duration: 37.999737619s FAIL (2025-09-23 14:10:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 37.999768969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...............................X...XX......X......] duration: 10.801441207s FAIL (2025-09-23 14:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.801480372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...............................X...XX......X.....] duration: 11.909717156s FAIL (2025-09-23 14:08:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.909750148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X......X...........] duration: 6.838071555s FAIL (2025-09-23 14:08:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.838098226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...............................X...XX......X....] duration: 7.546875276s FAIL (2025-09-23 14:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.546905813s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X......X..........] duration: 8.012600062s FAIL (2025-09-23 14:07:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 8.012638555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X...............................X...XX......X...] duration: 6.901249905s FAIL (2025-09-23 14:07:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.901281876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X......X.........] duration: 6.883017928s FAIL (2025-09-23 14:07:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.883052996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X...............................X...XX......X..] duration: 6.855227011s FAIL (2025-09-23 14:07:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.855270984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X......X........] duration: 6.559360066s FAIL (2025-09-23 14:06:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.559399341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X...............................X...XX......X.] duration: 7.101596984s FAIL (2025-09-23 14:06:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.101630839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X......X.......] duration: 6.327837234s FAIL (2025-09-23 14:06:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.327878051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X......X......] duration: 16.152766882s FAIL (2025-09-23 14:05:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 16.152798271s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)