Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX..XXX...X...XX...XX......X.X........X......X..X..........] duration: 17.840355872s FAIL (2025-09-08 15:37:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 17.840389446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.XX..XXX...X...XX...XX......X.X........X......X..X.........] duration: 6.814092187s FAIL (2025-09-08 15:36:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 6.814119098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.XX..XXX...X...XX...XX......X.X........X......X..X........] duration: 8.048206616s FAIL (2025-09-08 15:36:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 8.0482345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.XX..XXX...X...XX...XX......X.X........X......X..X.......] duration: 19.048049065s FAIL (2025-09-08 15:35:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 19.048080254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.XX..XXX...X...XX...XX......X.X........X......X..X......] duration: 16.767738505s FAIL (2025-09-08 15:34:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 16.767758253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.XX..XXX...X...XX...XX......X.X........X......X..X.....] duration: 7.494377523s FAIL (2025-09-08 15:33:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 7.494403633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X.XX..XXX...X...XX...XX......X.X........X......X..X....] duration: 8.179173771s FAIL (2025-09-08 15:33:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 8.179206444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.XX..XXX...X...XX...XX......X.X........X......X..X...] duration: 10.144486625s FAIL (2025-09-08 15:32:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 10.14451514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X.XX..XXX...X...XX...XX......X.X........X......X..X..] duration: 13.112267443s FAIL (2025-09-08 15:32:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 13.112296958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.......X.XX..XXX...X...XX...XX......X.X........X......X..X.] duration: 6.379987502s FAIL (2025-09-08 15:31:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 6.380013963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.......X.XX..XXX...X...XX...XX......X.X........X......X..X] duration: 30.003206388s FAIL (2025-09-08 15:31:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:30:32.666614807 (50h2m8.706187927s) [upload=43961]: Starting upload Sep 8 15:30:32.666790043 (50h2m8.706363183s) [upload=43961]: Done starting upload Sep 8 15:30:32.666857662 (50h2m8.706430792s) [upload=43961]: Uploading segments... Sep 8 15:30:32.667072032 (50h2m8.706645162s) [upload=43961]: Got next segment. Inline: false Sep 8 15:30:32.667123961 (50h2m8.706697091s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:30:32.667130023 (50h2m8.706703153s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:30:32.936736781 (50h2m8.976309911s) [seg_pos={0 0}, upload=43961]: Begin upload segment... Sep 8 15:30:32.936806293 (50h2m8.976379424s) [seg_pos={0 0}, upload=43961]: Done begin upload segment. Sep 8 15:30:32.936822826 (50h2m8.976395946s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.936835961 (50h2m8.976409081s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937276514 (50h2m8.976849634s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937316460 (50h2m8.97688958s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937596235 (50h2m8.977169355s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937661972 (50h2m8.977235092s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.937876893 (50h2m8.977450013s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938034544 (50h2m8.977607664s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938156378 (50h2m8.977729498s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938371780 (50h2m8.9779449s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.938805901 (50h2m8.978379021s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939087990 (50h2m8.97866111s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939430555 (50h2m8.979003675s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.939729617 (50h2m8.979302737s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940040282 (50h2m8.979613403s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:32.940315830 (50h2m8.97988895s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Uploading piece... Sep 8 15:30:34.769885049 (50h2m10.809458189s) [upload=43961]: Next returned nil segment Sep 8 15:30:34.769910377 (50h2m10.809483497s) [upload=43961]: Waiting for error group managing segments... Sep 8 15:30:35.106202521 (50h2m11.145775641s) [noise=true , address=45.91.116.12:4227 , piece_id=BLANWXAWLCCNDZG5MYPHJSA2MNDJPJJSMACA23KXBZGSEDIFO27Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135183518 (50h2m11.174756668s) [noise=true , address=45.91.116.6:4973 , piece_id=ZG6E56EACUEUJQUCMODOBXPF32WZE3526SWN4N6Q5JQA2QJKWPXQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.135342322 (50h2m11.174915442s) [noise=true , address=77.90.31.12:4590 , piece_id=JX7XUPDG32MRIITZ5GUNCLXEF7AMUH77CVQBUKD366XNFSBMLPCQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.142996298 (50h2m11.182569418s) [noise=true , address=145.79.227.5:4223 , piece_id=ZUDGIXHZK5OQNYVIHA7URY3M6RAGCGNS36OKWCL5MSVNMWG6DUFA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.145894113 (50h2m11.185467273s) [noise=true , address=45.91.116.5:4299 , piece_id=PE7VFMIKT72XMMJUFUDJIGCX2L47EJ3VI57GV5QNW24B4MUQHP6Q, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.147311355 (50h2m11.186884465s) [noise=true , address=45.91.116.13:4154 , piece_id=FEIMWSVA6L6MWHDYHN7NMF3NV6E77XXVEKOONUQFQ73SKALRFMKA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.155347032 (50h2m11.194920142s) [noise=true , address=45.91.116.9:4263 , piece_id=WLVJ4YTAOGDUABCYKQCNAMGVYOD2KRBGBZTLIGYBOXVE6KRXY6JA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.215238287 (50h2m11.254811407s) [noise=true , address=77.90.31.14:4663 , piece_id=BC2E7PN7CJ3CQSD4BLGDJZFGS5OSJXG4CWOSEDBSBPT2WNSYTZHA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.224811666 (50h2m11.264384786s) [noise=true , address=45.91.116.7:4009 , piece_id=QZOBXEE7LUY5EBLGGSXTYIWVZAGQ6DTPCFKT5UUVNTBC455ZDENA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.232391270 (50h2m11.27196439s) [noise=true , address=45.91.116.5:4297 , piece_id=IKETX2YQ6LMSALEL7FX52FBKDPGJ6XZAFJ4WHF4WINBFK3RH7M6A, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.257108799 (50h2m11.296681919s) [noise=true , address=45.91.116.14:4189 , piece_id=7WITNICFQEODIVZ27OH6K2X4HXABXHUMXJK4G37A7DWZ2QDT7VHQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.284248261 (50h2m11.323821381s) [noise=true , address=145.79.227.7:4293 , piece_id=EOL2OBAJMIEGJRCMS4LB5UJFYVLH3E7B3JMRK3WOVZOSPWMWKPNQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451413392 (50h2m11.490986542s) [noise=true , address=145.79.227.19:4077 , piece_id=4IJZXM7KFPMOJWG7UHU4ZG63WS45JBU7CGTKV6MURDD5D4MHGDWA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: <nil> Sep 8 15:30:35.451443259 (50h2m11.491016409s) [seg_pos={0 0}, upload=43961]: Segment reached optimal threshold of 13 pieces. Sep 8 15:30:35.451608896 (50h2m11.491182016s) [noise=true , address=45.91.116.4:4938 , piece_id=OZDKT3VO44D5RVZVENM2PXBL5Z6XUVWCSDNJCXR53A2AQ45HX5IA, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451650846 (50h2m11.491223976s) [noise=true , address=145.79.227.13:4803 , piece_id=C7XCY2GMNSE2BNJPADZ3KHPHEVUNXFM4TP4MYGI4M7XPHJFFOBAQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:30:35.451769984 (50h2m11.491343104s) [noise=true , address=77.90.31.15:4698 , piece_id=W5HA6H6QDQJIPDN3VI44A2FIBXKDXC7XOAKNQVPY5OZ4ZIHKPZTQ, seg_pos={0 0}, upload=43961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:51502->77.90.31.15:4698: use of closed network connection Sep 8 15:30:35.451811513 (50h2m11.491384633s) [seg_pos={0 0}, upload=43961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:30:35.451903901 (50h2m11.491477021s) [upload=43961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:30:35.451908299 (50h2m11.491481419s) [upload=43961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:30:35.782179801 (50h2m11.821752921s) [upload=43961]: Done uploading segments... Sep 8 15:30:35.782203456 (50h2m11.821776576s) [upload=43961]: Upload finished. err: <nil> Duration: 30.003236006s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.......X.XX..XXX...X...XX...XX......X.X........X......X..] duration: 14.606969522s FAIL (2025-09-08 15:29:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:28:17.248541395 (49h59m53.288114515s) [upload=43928]: Starting upload Sep 8 15:28:17.248612612 (49h59m53.288185732s) [upload=43928]: Done starting upload Sep 8 15:28:17.248732160 (49h59m53.28830528s) [upload=43928]: Uploading segments... Sep 8 15:28:17.248834737 (49h59m53.288407857s) [upload=43928]: Got next segment. Inline: false Sep 8 15:28:17.248863282 (49h59m53.288436392s) [upload=43928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:28:17.248867530 (49h59m53.28844065s) [upload=43928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:28:17.514744169 (49h59m53.554317289s) [seg_pos={0 0}, upload=43928]: Begin upload segment... Sep 8 15:28:17.514835033 (49h59m53.554408153s) [seg_pos={0 0}, upload=43928]: Done begin upload segment. Sep 8 15:28:17.514850202 (49h59m53.554423322s) [noise=true , address=145.79.227.16:4909 , piece_id=YNHBCRJOC3DKJRJ4NKDBZW6RMLML3K3V45FI7T54QSQPBPUII67A, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.514860843 (49h59m53.554433963s) [noise=true , address=45.91.116.14:4189 , piece_id=GL7LXVUOKGDISQZP4DA347BWHA6RMCJOR6IXZCG6NAB7OCMNGPYA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.514908022 (49h59m53.554481142s) [noise=true , address=77.90.31.12:4589 , piece_id=F4BFUA3B4W5JBAFY3IROWLX2B52T4JKRQ6BPFCOC4VVQ3CFTK4TA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515215491 (49h59m53.554788611s) [noise=true , address=45.91.116.8:4048 , piece_id=64KBGB5OQW2IKLF2Q4HKPV7NNQETXCSK32YJNKFQJPAS7NR6HWDQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515486109 (49h59m53.555059229s) [noise=true , address=45.91.116.9:4262 , piece_id=G4YK4CT3YNVS3NXZO4G6K3DSABWFQBZKF6SKB62SHS5F4KCSTNFQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515749002 (49h59m53.555322122s) [noise=true , address=77.90.31.12:4592 , piece_id=VJOLOHPFYIABUYYITGSPM4QTGDIDSSIWMUZMDNAZ2BJQPFIIT4VQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516012055 (49h59m53.555585175s) [noise=true , address=145.79.227.18:4006 , piece_id=2CARIS6PYG6NJKLFMKWP57PVRXSL64XD3B27F2K4OQA5GD623FAQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516247866 (49h59m53.555820986s) [noise=true , address=45.91.116.10:4082 , piece_id=5GGWTNV4MB6AZLUPKSUP6JUM42MD4L6ZJWTPWYR5PQYVAZOIYYQQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516460172 (49h59m53.556033292s) [noise=true , address=77.90.31.21:4879 , piece_id=ONINPPSJBCYMGTYEMIWFXLPLKQTX5CEKZ2RYG4JYXAVKFIUQGPKA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516674112 (49h59m53.556247232s) [noise=true , address=77.90.31.18:4806 , piece_id=TIHQBDTR7QZG3VEUM4OWG5ZIP3UAIBGX4ND6DM3YU64FJFP434HQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516885316 (49h59m53.556458436s) [noise=true , address=45.91.116.11:4118 , piece_id=H4BLBGUECHPVEB2LZW3L3QC3AQ7ULLRECSG66UCJ6KDAX4FOG3IQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517107161 (49h59m53.556680281s) [noise=true , address=145.79.227.11:4731 , piece_id=6RSUXXDSQFA34667KSTDJWCG7AVTK2TF5QLDS62F3SHAJR6UFHAQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517334746 (49h59m53.556907856s) [noise=true , address=145.79.227.6:4259 , piece_id=OCJMDGJEEEPIST6TRPZZJXD2RTFAYRZT54ZZ5AI7UALLGMY6TKPQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517537775 (49h59m53.557110895s) [noise=true , address=45.91.116.4:4940 , piece_id=LFW4N2P6NHAV3WPVC7QDT7UZZGBERHNXBWYYMXGRISSX4CGOJO2Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.518197487 (49h59m53.557770607s) [noise=true , address=45.91.116.9:4261 , piece_id=XKH35PD2BLNMJSBZQC3L6JQOETDWEXKFVRJP2G4TXY7FHL4AOS2Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.518721289 (49h59m53.558294409s) [noise=true , address=145.79.227.19:4078 , piece_id=6HB47K6SFM4YSL4KGCIMD3RMX5YVXYUFZIPRMSWLKPEGEB54IS3Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:19.352260678 (49h59m55.391833798s) [upload=43928]: Next returned nil segment Sep 8 15:28:19.352282699 (49h59m55.391855819s) [upload=43928]: Waiting for error group managing segments... Sep 8 15:28:19.649473881 (49h59m55.689047001s) [noise=true , address=45.91.116.9:4261 , piece_id=XKH35PD2BLNMJSBZQC3L6JQOETDWEXKFVRJP2G4TXY7FHL4AOS2Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.667325100 (49h59m55.70689822s) [noise=true , address=45.91.116.9:4262 , piece_id=G4YK4CT3YNVS3NXZO4G6K3DSABWFQBZKF6SKB62SHS5F4KCSTNFQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.687314391 (49h59m55.726887521s) [noise=true , address=45.91.116.4:4940 , piece_id=LFW4N2P6NHAV3WPVC7QDT7UZZGBERHNXBWYYMXGRISSX4CGOJO2Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.697023009 (49h59m55.736596139s) [noise=true , address=45.91.116.10:4082 , piece_id=5GGWTNV4MB6AZLUPKSUP6JUM42MD4L6ZJWTPWYR5PQYVAZOIYYQQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.727572377 (49h59m55.767145497s) [noise=true , address=145.79.227.19:4078 , piece_id=6HB47K6SFM4YSL4KGCIMD3RMX5YVXYUFZIPRMSWLKPEGEB54IS3Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.751945486 (49h59m55.791518606s) [noise=true , address=145.79.227.16:4909 , piece_id=YNHBCRJOC3DKJRJ4NKDBZW6RMLML3K3V45FI7T54QSQPBPUII67A, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.768161075 (49h59m55.807734195s) [noise=true , address=77.90.31.12:4589 , piece_id=F4BFUA3B4W5JBAFY3IROWLX2B52T4JKRQ6BPFCOC4VVQ3CFTK4TA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.776992545 (49h59m55.816565665s) [noise=true , address=77.90.31.18:4806 , piece_id=TIHQBDTR7QZG3VEUM4OWG5ZIP3UAIBGX4ND6DM3YU64FJFP434HQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.781342519 (49h59m55.820915649s) [noise=true , address=45.91.116.11:4118 , piece_id=H4BLBGUECHPVEB2LZW3L3QC3AQ7ULLRECSG66UCJ6KDAX4FOG3IQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824500324 (49h59m55.864073444s) [noise=true , address=77.90.31.21:4879 , piece_id=ONINPPSJBCYMGTYEMIWFXLPLKQTX5CEKZ2RYG4JYXAVKFIUQGPKA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824796100 (49h59m55.86436922s) [noise=true , address=45.91.116.8:4048 , piece_id=64KBGB5OQW2IKLF2Q4HKPV7NNQETXCSK32YJNKFQJPAS7NR6HWDQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824942139 (49h59m55.86451531s) [noise=true , address=145.79.227.6:4259 , piece_id=OCJMDGJEEEPIST6TRPZZJXD2RTFAYRZT54ZZ5AI7UALLGMY6TKPQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.838877305 (49h59m55.878450425s) [noise=true , address=45.91.116.14:4189 , piece_id=GL7LXVUOKGDISQZP4DA347BWHA6RMCJOR6IXZCG6NAB7OCMNGPYA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.838896191 (49h59m55.878469301s) [seg_pos={0 0}, upload=43928]: Segment reached optimal threshold of 13 pieces. Sep 8 15:28:19.839141170 (49h59m55.87871429s) [noise=true , address=77.90.31.12:4592 , piece_id=VJOLOHPFYIABUYYITGSPM4QTGDIDSSIWMUZMDNAZ2BJQPFIIT4VQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection Sep 8 15:28:19.839155096 (49h59m55.878728216s) [noise=true , address=145.79.227.18:4006 , piece_id=2CARIS6PYG6NJKLFMKWP57PVRXSL64XD3B27F2K4OQA5GD623FAQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Sep 8 15:28:19.839218228 (49h59m55.878791348s) [noise=true , address=145.79.227.11:4731 , piece_id=6RSUXXDSQFA34667KSTDJWCG7AVTK2TF5QLDS62F3SHAJR6UFHAQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Sep 8 15:28:19.839235841 (49h59m55.878808961s) [seg_pos={0 0}, upload=43928]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:28:19.839300976 (49h59m55.878874086s) [upload=43928]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:28:19.839304913 (49h59m55.878878033s) [upload=43928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:28:20.178732216 (49h59m56.218305346s) [upload=43928]: Done uploading segments... Sep 8 15:28:20.178752685 (49h59m56.218325805s) [upload=43928]: Upload finished. err: <nil> Duration: 14.606990933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXX.......X.XX..XXX...X...XX...XX......X.X........X......X.] duration: 7.676930016s FAIL (2025-09-08 15:29:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:28:17.248541395 (49h59m53.288114515s) [upload=43928]: Starting upload Sep 8 15:28:17.248612612 (49h59m53.288185732s) [upload=43928]: Done starting upload Sep 8 15:28:17.248732160 (49h59m53.28830528s) [upload=43928]: Uploading segments... Sep 8 15:28:17.248834737 (49h59m53.288407857s) [upload=43928]: Got next segment. Inline: false Sep 8 15:28:17.248863282 (49h59m53.288436392s) [upload=43928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:28:17.248867530 (49h59m53.28844065s) [upload=43928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:28:17.514744169 (49h59m53.554317289s) [seg_pos={0 0}, upload=43928]: Begin upload segment... Sep 8 15:28:17.514835033 (49h59m53.554408153s) [seg_pos={0 0}, upload=43928]: Done begin upload segment. Sep 8 15:28:17.514850202 (49h59m53.554423322s) [noise=true , address=145.79.227.16:4909 , piece_id=YNHBCRJOC3DKJRJ4NKDBZW6RMLML3K3V45FI7T54QSQPBPUII67A, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.514860843 (49h59m53.554433963s) [noise=true , address=45.91.116.14:4189 , piece_id=GL7LXVUOKGDISQZP4DA347BWHA6RMCJOR6IXZCG6NAB7OCMNGPYA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.514908022 (49h59m53.554481142s) [noise=true , address=77.90.31.12:4589 , piece_id=F4BFUA3B4W5JBAFY3IROWLX2B52T4JKRQ6BPFCOC4VVQ3CFTK4TA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515215491 (49h59m53.554788611s) [noise=true , address=45.91.116.8:4048 , piece_id=64KBGB5OQW2IKLF2Q4HKPV7NNQETXCSK32YJNKFQJPAS7NR6HWDQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515486109 (49h59m53.555059229s) [noise=true , address=45.91.116.9:4262 , piece_id=G4YK4CT3YNVS3NXZO4G6K3DSABWFQBZKF6SKB62SHS5F4KCSTNFQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515749002 (49h59m53.555322122s) [noise=true , address=77.90.31.12:4592 , piece_id=VJOLOHPFYIABUYYITGSPM4QTGDIDSSIWMUZMDNAZ2BJQPFIIT4VQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516012055 (49h59m53.555585175s) [noise=true , address=145.79.227.18:4006 , piece_id=2CARIS6PYG6NJKLFMKWP57PVRXSL64XD3B27F2K4OQA5GD623FAQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516247866 (49h59m53.555820986s) [noise=true , address=45.91.116.10:4082 , piece_id=5GGWTNV4MB6AZLUPKSUP6JUM42MD4L6ZJWTPWYR5PQYVAZOIYYQQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516460172 (49h59m53.556033292s) [noise=true , address=77.90.31.21:4879 , piece_id=ONINPPSJBCYMGTYEMIWFXLPLKQTX5CEKZ2RYG4JYXAVKFIUQGPKA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516674112 (49h59m53.556247232s) [noise=true , address=77.90.31.18:4806 , piece_id=TIHQBDTR7QZG3VEUM4OWG5ZIP3UAIBGX4ND6DM3YU64FJFP434HQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516885316 (49h59m53.556458436s) [noise=true , address=45.91.116.11:4118 , piece_id=H4BLBGUECHPVEB2LZW3L3QC3AQ7ULLRECSG66UCJ6KDAX4FOG3IQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517107161 (49h59m53.556680281s) [noise=true , address=145.79.227.11:4731 , piece_id=6RSUXXDSQFA34667KSTDJWCG7AVTK2TF5QLDS62F3SHAJR6UFHAQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517334746 (49h59m53.556907856s) [noise=true , address=145.79.227.6:4259 , piece_id=OCJMDGJEEEPIST6TRPZZJXD2RTFAYRZT54ZZ5AI7UALLGMY6TKPQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517537775 (49h59m53.557110895s) [noise=true , address=45.91.116.4:4940 , piece_id=LFW4N2P6NHAV3WPVC7QDT7UZZGBERHNXBWYYMXGRISSX4CGOJO2Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.518197487 (49h59m53.557770607s) [noise=true , address=45.91.116.9:4261 , piece_id=XKH35PD2BLNMJSBZQC3L6JQOETDWEXKFVRJP2G4TXY7FHL4AOS2Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.518721289 (49h59m53.558294409s) [noise=true , address=145.79.227.19:4078 , piece_id=6HB47K6SFM4YSL4KGCIMD3RMX5YVXYUFZIPRMSWLKPEGEB54IS3Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:19.352260678 (49h59m55.391833798s) [upload=43928]: Next returned nil segment Sep 8 15:28:19.352282699 (49h59m55.391855819s) [upload=43928]: Waiting for error group managing segments... Sep 8 15:28:19.649473881 (49h59m55.689047001s) [noise=true , address=45.91.116.9:4261 , piece_id=XKH35PD2BLNMJSBZQC3L6JQOETDWEXKFVRJP2G4TXY7FHL4AOS2Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.667325100 (49h59m55.70689822s) [noise=true , address=45.91.116.9:4262 , piece_id=G4YK4CT3YNVS3NXZO4G6K3DSABWFQBZKF6SKB62SHS5F4KCSTNFQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.687314391 (49h59m55.726887521s) [noise=true , address=45.91.116.4:4940 , piece_id=LFW4N2P6NHAV3WPVC7QDT7UZZGBERHNXBWYYMXGRISSX4CGOJO2Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.697023009 (49h59m55.736596139s) [noise=true , address=45.91.116.10:4082 , piece_id=5GGWTNV4MB6AZLUPKSUP6JUM42MD4L6ZJWTPWYR5PQYVAZOIYYQQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.727572377 (49h59m55.767145497s) [noise=true , address=145.79.227.19:4078 , piece_id=6HB47K6SFM4YSL4KGCIMD3RMX5YVXYUFZIPRMSWLKPEGEB54IS3Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.751945486 (49h59m55.791518606s) [noise=true , address=145.79.227.16:4909 , piece_id=YNHBCRJOC3DKJRJ4NKDBZW6RMLML3K3V45FI7T54QSQPBPUII67A, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.768161075 (49h59m55.807734195s) [noise=true , address=77.90.31.12:4589 , piece_id=F4BFUA3B4W5JBAFY3IROWLX2B52T4JKRQ6BPFCOC4VVQ3CFTK4TA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.776992545 (49h59m55.816565665s) [noise=true , address=77.90.31.18:4806 , piece_id=TIHQBDTR7QZG3VEUM4OWG5ZIP3UAIBGX4ND6DM3YU64FJFP434HQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.781342519 (49h59m55.820915649s) [noise=true , address=45.91.116.11:4118 , piece_id=H4BLBGUECHPVEB2LZW3L3QC3AQ7ULLRECSG66UCJ6KDAX4FOG3IQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824500324 (49h59m55.864073444s) [noise=true , address=77.90.31.21:4879 , piece_id=ONINPPSJBCYMGTYEMIWFXLPLKQTX5CEKZ2RYG4JYXAVKFIUQGPKA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824796100 (49h59m55.86436922s) [noise=true , address=45.91.116.8:4048 , piece_id=64KBGB5OQW2IKLF2Q4HKPV7NNQETXCSK32YJNKFQJPAS7NR6HWDQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824942139 (49h59m55.86451531s) [noise=true , address=145.79.227.6:4259 , piece_id=OCJMDGJEEEPIST6TRPZZJXD2RTFAYRZT54ZZ5AI7UALLGMY6TKPQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.838877305 (49h59m55.878450425s) [noise=true , address=45.91.116.14:4189 , piece_id=GL7LXVUOKGDISQZP4DA347BWHA6RMCJOR6IXZCG6NAB7OCMNGPYA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.838896191 (49h59m55.878469301s) [seg_pos={0 0}, upload=43928]: Segment reached optimal threshold of 13 pieces. Sep 8 15:28:19.839141170 (49h59m55.87871429s) [noise=true , address=77.90.31.12:4592 , piece_id=VJOLOHPFYIABUYYITGSPM4QTGDIDSSIWMUZMDNAZ2BJQPFIIT4VQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection Sep 8 15:28:19.839155096 (49h59m55.878728216s) [noise=true , address=145.79.227.18:4006 , piece_id=2CARIS6PYG6NJKLFMKWP57PVRXSL64XD3B27F2K4OQA5GD623FAQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Sep 8 15:28:19.839218228 (49h59m55.878791348s) [noise=true , address=145.79.227.11:4731 , piece_id=6RSUXXDSQFA34667KSTDJWCG7AVTK2TF5QLDS62F3SHAJR6UFHAQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Sep 8 15:28:19.839235841 (49h59m55.878808961s) [seg_pos={0 0}, upload=43928]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:28:19.839300976 (49h59m55.878874086s) [upload=43928]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:28:19.839304913 (49h59m55.878878033s) [upload=43928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:28:20.178732216 (49h59m56.218305346s) [upload=43928]: Done uploading segments... Sep 8 15:28:20.178752685 (49h59m56.218325805s) [upload=43928]: Upload finished. err: <nil> Duration: 7.676957147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXX.......X.XX..XXX...X...XX...XX......X.X........X......X] duration: 30.001379974s FAIL (2025-09-08 15:28:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:28:17.248541395 (49h59m53.288114515s) [upload=43928]: Starting upload Sep 8 15:28:17.248612612 (49h59m53.288185732s) [upload=43928]: Done starting upload Sep 8 15:28:17.248732160 (49h59m53.28830528s) [upload=43928]: Uploading segments... Sep 8 15:28:17.248834737 (49h59m53.288407857s) [upload=43928]: Got next segment. Inline: false Sep 8 15:28:17.248863282 (49h59m53.288436392s) [upload=43928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:28:17.248867530 (49h59m53.28844065s) [upload=43928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:28:17.514744169 (49h59m53.554317289s) [seg_pos={0 0}, upload=43928]: Begin upload segment... Sep 8 15:28:17.514835033 (49h59m53.554408153s) [seg_pos={0 0}, upload=43928]: Done begin upload segment. Sep 8 15:28:17.514850202 (49h59m53.554423322s) [noise=true , address=145.79.227.16:4909 , piece_id=YNHBCRJOC3DKJRJ4NKDBZW6RMLML3K3V45FI7T54QSQPBPUII67A, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.514860843 (49h59m53.554433963s) [noise=true , address=45.91.116.14:4189 , piece_id=GL7LXVUOKGDISQZP4DA347BWHA6RMCJOR6IXZCG6NAB7OCMNGPYA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.514908022 (49h59m53.554481142s) [noise=true , address=77.90.31.12:4589 , piece_id=F4BFUA3B4W5JBAFY3IROWLX2B52T4JKRQ6BPFCOC4VVQ3CFTK4TA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515215491 (49h59m53.554788611s) [noise=true , address=45.91.116.8:4048 , piece_id=64KBGB5OQW2IKLF2Q4HKPV7NNQETXCSK32YJNKFQJPAS7NR6HWDQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515486109 (49h59m53.555059229s) [noise=true , address=45.91.116.9:4262 , piece_id=G4YK4CT3YNVS3NXZO4G6K3DSABWFQBZKF6SKB62SHS5F4KCSTNFQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.515749002 (49h59m53.555322122s) [noise=true , address=77.90.31.12:4592 , piece_id=VJOLOHPFYIABUYYITGSPM4QTGDIDSSIWMUZMDNAZ2BJQPFIIT4VQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516012055 (49h59m53.555585175s) [noise=true , address=145.79.227.18:4006 , piece_id=2CARIS6PYG6NJKLFMKWP57PVRXSL64XD3B27F2K4OQA5GD623FAQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516247866 (49h59m53.555820986s) [noise=true , address=45.91.116.10:4082 , piece_id=5GGWTNV4MB6AZLUPKSUP6JUM42MD4L6ZJWTPWYR5PQYVAZOIYYQQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516460172 (49h59m53.556033292s) [noise=true , address=77.90.31.21:4879 , piece_id=ONINPPSJBCYMGTYEMIWFXLPLKQTX5CEKZ2RYG4JYXAVKFIUQGPKA, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516674112 (49h59m53.556247232s) [noise=true , address=77.90.31.18:4806 , piece_id=TIHQBDTR7QZG3VEUM4OWG5ZIP3UAIBGX4ND6DM3YU64FJFP434HQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.516885316 (49h59m53.556458436s) [noise=true , address=45.91.116.11:4118 , piece_id=H4BLBGUECHPVEB2LZW3L3QC3AQ7ULLRECSG66UCJ6KDAX4FOG3IQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517107161 (49h59m53.556680281s) [noise=true , address=145.79.227.11:4731 , piece_id=6RSUXXDSQFA34667KSTDJWCG7AVTK2TF5QLDS62F3SHAJR6UFHAQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517334746 (49h59m53.556907856s) [noise=true , address=145.79.227.6:4259 , piece_id=OCJMDGJEEEPIST6TRPZZJXD2RTFAYRZT54ZZ5AI7UALLGMY6TKPQ, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.517537775 (49h59m53.557110895s) [noise=true , address=45.91.116.4:4940 , piece_id=LFW4N2P6NHAV3WPVC7QDT7UZZGBERHNXBWYYMXGRISSX4CGOJO2Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.518197487 (49h59m53.557770607s) [noise=true , address=45.91.116.9:4261 , piece_id=XKH35PD2BLNMJSBZQC3L6JQOETDWEXKFVRJP2G4TXY7FHL4AOS2Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:17.518721289 (49h59m53.558294409s) [noise=true , address=145.79.227.19:4078 , piece_id=6HB47K6SFM4YSL4KGCIMD3RMX5YVXYUFZIPRMSWLKPEGEB54IS3Q, seg_pos={0 0}, upload=43928]: Uploading piece... Sep 8 15:28:19.352260678 (49h59m55.391833798s) [upload=43928]: Next returned nil segment Sep 8 15:28:19.352282699 (49h59m55.391855819s) [upload=43928]: Waiting for error group managing segments... Sep 8 15:28:19.649473881 (49h59m55.689047001s) [noise=true , address=45.91.116.9:4261 , piece_id=XKH35PD2BLNMJSBZQC3L6JQOETDWEXKFVRJP2G4TXY7FHL4AOS2Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.667325100 (49h59m55.70689822s) [noise=true , address=45.91.116.9:4262 , piece_id=G4YK4CT3YNVS3NXZO4G6K3DSABWFQBZKF6SKB62SHS5F4KCSTNFQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.687314391 (49h59m55.726887521s) [noise=true , address=45.91.116.4:4940 , piece_id=LFW4N2P6NHAV3WPVC7QDT7UZZGBERHNXBWYYMXGRISSX4CGOJO2Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.697023009 (49h59m55.736596139s) [noise=true , address=45.91.116.10:4082 , piece_id=5GGWTNV4MB6AZLUPKSUP6JUM42MD4L6ZJWTPWYR5PQYVAZOIYYQQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.727572377 (49h59m55.767145497s) [noise=true , address=145.79.227.19:4078 , piece_id=6HB47K6SFM4YSL4KGCIMD3RMX5YVXYUFZIPRMSWLKPEGEB54IS3Q, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.751945486 (49h59m55.791518606s) [noise=true , address=145.79.227.16:4909 , piece_id=YNHBCRJOC3DKJRJ4NKDBZW6RMLML3K3V45FI7T54QSQPBPUII67A, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.768161075 (49h59m55.807734195s) [noise=true , address=77.90.31.12:4589 , piece_id=F4BFUA3B4W5JBAFY3IROWLX2B52T4JKRQ6BPFCOC4VVQ3CFTK4TA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.776992545 (49h59m55.816565665s) [noise=true , address=77.90.31.18:4806 , piece_id=TIHQBDTR7QZG3VEUM4OWG5ZIP3UAIBGX4ND6DM3YU64FJFP434HQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.781342519 (49h59m55.820915649s) [noise=true , address=45.91.116.11:4118 , piece_id=H4BLBGUECHPVEB2LZW3L3QC3AQ7ULLRECSG66UCJ6KDAX4FOG3IQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824500324 (49h59m55.864073444s) [noise=true , address=77.90.31.21:4879 , piece_id=ONINPPSJBCYMGTYEMIWFXLPLKQTX5CEKZ2RYG4JYXAVKFIUQGPKA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824796100 (49h59m55.86436922s) [noise=true , address=45.91.116.8:4048 , piece_id=64KBGB5OQW2IKLF2Q4HKPV7NNQETXCSK32YJNKFQJPAS7NR6HWDQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.824942139 (49h59m55.86451531s) [noise=true , address=145.79.227.6:4259 , piece_id=OCJMDGJEEEPIST6TRPZZJXD2RTFAYRZT54ZZ5AI7UALLGMY6TKPQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.838877305 (49h59m55.878450425s) [noise=true , address=45.91.116.14:4189 , piece_id=GL7LXVUOKGDISQZP4DA347BWHA6RMCJOR6IXZCG6NAB7OCMNGPYA, seg_pos={0 0}, upload=43928]: Done uploading piece. err: <nil> Sep 8 15:28:19.838896191 (49h59m55.878469301s) [seg_pos={0 0}, upload=43928]: Segment reached optimal threshold of 13 pieces. Sep 8 15:28:19.839141170 (49h59m55.87871429s) [noise=true , address=77.90.31.12:4592 , piece_id=VJOLOHPFYIABUYYITGSPM4QTGDIDSSIWMUZMDNAZ2BJQPFIIT4VQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35852->77.90.31.12:4592: use of closed network connection Sep 8 15:28:19.839155096 (49h59m55.878728216s) [noise=true , address=145.79.227.18:4006 , piece_id=2CARIS6PYG6NJKLFMKWP57PVRXSL64XD3B27F2K4OQA5GD623FAQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Sep 8 15:28:19.839218228 (49h59m55.878791348s) [noise=true , address=145.79.227.11:4731 , piece_id=6RSUXXDSQFA34667KSTDJWCG7AVTK2TF5QLDS62F3SHAJR6UFHAQ, seg_pos={0 0}, upload=43928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Sep 8 15:28:19.839235841 (49h59m55.878808961s) [seg_pos={0 0}, upload=43928]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:28:19.839300976 (49h59m55.878874086s) [upload=43928]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:28:19.839304913 (49h59m55.878878033s) [upload=43928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:28:20.178732216 (49h59m56.218305346s) [upload=43928]: Done uploading segments... Sep 8 15:28:20.178752685 (49h59m56.218325805s) [upload=43928]: Upload finished. err: <nil> Duration: 30.001405753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXX.......X.XX..XXX...X...XX...XX......X.X........X......] duration: 17.155949975s FAIL (2025-09-08 15:27:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:22:47.727033066 (49h54m23.766606187s) [upload=43849]: Starting upload Sep 8 15:22:47.727221938 (49h54m23.766795068s) [upload=43849]: Done starting upload Sep 8 15:22:47.727349863 (49h54m23.766922983s) [upload=43849]: Uploading segments... Sep 8 15:22:47.727494449 (49h54m23.76706757s) [upload=43849]: Got next segment. Inline: false Sep 8 15:22:47.727525218 (49h54m23.767098338s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:22:47.727529857 (49h54m23.767102977s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:22:47.993405316 (49h54m24.032978436s) [seg_pos={0 0}, upload=43849]: Begin upload segment... Sep 8 15:22:47.993452405 (49h54m24.033025525s) [seg_pos={0 0}, upload=43849]: Done begin upload segment. Sep 8 15:22:47.993477513 (49h54m24.033050633s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993483054 (49h54m24.033056174s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993861548 (49h54m24.033434678s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994021384 (49h54m24.033594514s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994387685 (49h54m24.033960815s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994688891 (49h54m24.034262011s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994751952 (49h54m24.034325072s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994922869 (49h54m24.034495989s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995228193 (49h54m24.034801313s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995337603 (49h54m24.034910723s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995487019 (49h54m24.035060139s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995661993 (49h54m24.035235113s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995782233 (49h54m24.035355353s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995935306 (49h54m24.035508426s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996023996 (49h54m24.035597116s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996149997 (49h54m24.035723117s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:48.253260102 (49h54m24.292833222s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57090->77.90.31.16:4735: write: connection refused; conn closed: op Sep 8 15:22:48.275865279 (49h54m24.315438399s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45466->77.90.31.16:4733: write: connection refused; conn closed: op Sep 8 15:22:49.850841260 (49h54m25.89041439s) [upload=43849]: Next returned nil segment Sep 8 15:22:49.850865206 (49h54m25.890438356s) [upload=43849]: Waiting for error group managing segments... Sep 8 15:22:50.175466458 (49h54m26.215039578s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.187812522 (49h54m26.227385642s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.195784016 (49h54m26.235357136s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.201113334 (49h54m26.240686454s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.209776070 (49h54m26.24934919s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.228795255 (49h54m26.268368375s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.232605095 (49h54m26.272178215s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.233700721 (49h54m26.273273831s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.290809673 (49h54m26.330382803s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.298656217 (49h54m26.338229337s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.307520770 (49h54m26.34709389s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.323901867 (49h54m26.363474987s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342940047 (49h54m26.382513197s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342987949 (49h54m26.382561069s) [seg_pos={0 0}, upload=43849]: Segment reached optimal threshold of 13 pieces. Sep 8 15:22:50.343193583 (49h54m26.382766703s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:22:50.343221546 (49h54m26.382794666s) [seg_pos={0 0}, upload=43849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:22:50.343287392 (49h54m26.382860512s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:22:50.343292141 (49h54m26.382865261s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:22:50.669106604 (49h54m26.708679724s) [upload=43849]: Done uploading segments... Sep 8 15:22:50.669125721 (49h54m26.708698841s) [upload=43849]: Upload finished. err: <nil> Duration: 17.155969833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XXXXXX.......X.XX..XXX...X...XX...XX......X.X........X.....] duration: 7.324088854s FAIL (2025-09-08 15:27:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:22:47.727033066 (49h54m23.766606187s) [upload=43849]: Starting upload Sep 8 15:22:47.727221938 (49h54m23.766795068s) [upload=43849]: Done starting upload Sep 8 15:22:47.727349863 (49h54m23.766922983s) [upload=43849]: Uploading segments... Sep 8 15:22:47.727494449 (49h54m23.76706757s) [upload=43849]: Got next segment. Inline: false Sep 8 15:22:47.727525218 (49h54m23.767098338s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:22:47.727529857 (49h54m23.767102977s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:22:47.993405316 (49h54m24.032978436s) [seg_pos={0 0}, upload=43849]: Begin upload segment... Sep 8 15:22:47.993452405 (49h54m24.033025525s) [seg_pos={0 0}, upload=43849]: Done begin upload segment. Sep 8 15:22:47.993477513 (49h54m24.033050633s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993483054 (49h54m24.033056174s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993861548 (49h54m24.033434678s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994021384 (49h54m24.033594514s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994387685 (49h54m24.033960815s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994688891 (49h54m24.034262011s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994751952 (49h54m24.034325072s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994922869 (49h54m24.034495989s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995228193 (49h54m24.034801313s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995337603 (49h54m24.034910723s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995487019 (49h54m24.035060139s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995661993 (49h54m24.035235113s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995782233 (49h54m24.035355353s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995935306 (49h54m24.035508426s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996023996 (49h54m24.035597116s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996149997 (49h54m24.035723117s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:48.253260102 (49h54m24.292833222s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57090->77.90.31.16:4735: write: connection refused; conn closed: op Sep 8 15:22:48.275865279 (49h54m24.315438399s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45466->77.90.31.16:4733: write: connection refused; conn closed: op Sep 8 15:22:49.850841260 (49h54m25.89041439s) [upload=43849]: Next returned nil segment Sep 8 15:22:49.850865206 (49h54m25.890438356s) [upload=43849]: Waiting for error group managing segments... Sep 8 15:22:50.175466458 (49h54m26.215039578s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.187812522 (49h54m26.227385642s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.195784016 (49h54m26.235357136s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.201113334 (49h54m26.240686454s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.209776070 (49h54m26.24934919s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.228795255 (49h54m26.268368375s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.232605095 (49h54m26.272178215s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.233700721 (49h54m26.273273831s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.290809673 (49h54m26.330382803s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.298656217 (49h54m26.338229337s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.307520770 (49h54m26.34709389s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.323901867 (49h54m26.363474987s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342940047 (49h54m26.382513197s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342987949 (49h54m26.382561069s) [seg_pos={0 0}, upload=43849]: Segment reached optimal threshold of 13 pieces. Sep 8 15:22:50.343193583 (49h54m26.382766703s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:22:50.343221546 (49h54m26.382794666s) [seg_pos={0 0}, upload=43849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:22:50.343287392 (49h54m26.382860512s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:22:50.343292141 (49h54m26.382865261s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:22:50.669106604 (49h54m26.708679724s) [upload=43849]: Done uploading segments... Sep 8 15:22:50.669125721 (49h54m26.708698841s) [upload=43849]: Upload finished. err: <nil> Duration: 7.32411804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XXXXXX.......X.XX..XXX...X...XX...XX......X.X........X....] duration: 9.300039331s FAIL (2025-09-08 15:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:22:47.727033066 (49h54m23.766606187s) [upload=43849]: Starting upload Sep 8 15:22:47.727221938 (49h54m23.766795068s) [upload=43849]: Done starting upload Sep 8 15:22:47.727349863 (49h54m23.766922983s) [upload=43849]: Uploading segments... Sep 8 15:22:47.727494449 (49h54m23.76706757s) [upload=43849]: Got next segment. Inline: false Sep 8 15:22:47.727525218 (49h54m23.767098338s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:22:47.727529857 (49h54m23.767102977s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:22:47.993405316 (49h54m24.032978436s) [seg_pos={0 0}, upload=43849]: Begin upload segment... Sep 8 15:22:47.993452405 (49h54m24.033025525s) [seg_pos={0 0}, upload=43849]: Done begin upload segment. Sep 8 15:22:47.993477513 (49h54m24.033050633s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993483054 (49h54m24.033056174s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993861548 (49h54m24.033434678s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994021384 (49h54m24.033594514s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994387685 (49h54m24.033960815s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994688891 (49h54m24.034262011s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994751952 (49h54m24.034325072s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994922869 (49h54m24.034495989s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995228193 (49h54m24.034801313s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995337603 (49h54m24.034910723s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995487019 (49h54m24.035060139s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995661993 (49h54m24.035235113s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995782233 (49h54m24.035355353s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995935306 (49h54m24.035508426s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996023996 (49h54m24.035597116s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996149997 (49h54m24.035723117s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:48.253260102 (49h54m24.292833222s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57090->77.90.31.16:4735: write: connection refused; conn closed: op Sep 8 15:22:48.275865279 (49h54m24.315438399s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45466->77.90.31.16:4733: write: connection refused; conn closed: op Sep 8 15:22:49.850841260 (49h54m25.89041439s) [upload=43849]: Next returned nil segment Sep 8 15:22:49.850865206 (49h54m25.890438356s) [upload=43849]: Waiting for error group managing segments... Sep 8 15:22:50.175466458 (49h54m26.215039578s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.187812522 (49h54m26.227385642s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.195784016 (49h54m26.235357136s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.201113334 (49h54m26.240686454s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.209776070 (49h54m26.24934919s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.228795255 (49h54m26.268368375s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.232605095 (49h54m26.272178215s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.233700721 (49h54m26.273273831s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.290809673 (49h54m26.330382803s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.298656217 (49h54m26.338229337s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.307520770 (49h54m26.34709389s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.323901867 (49h54m26.363474987s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342940047 (49h54m26.382513197s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342987949 (49h54m26.382561069s) [seg_pos={0 0}, upload=43849]: Segment reached optimal threshold of 13 pieces. Sep 8 15:22:50.343193583 (49h54m26.382766703s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:22:50.343221546 (49h54m26.382794666s) [seg_pos={0 0}, upload=43849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:22:50.343287392 (49h54m26.382860512s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:22:50.343292141 (49h54m26.382865261s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:22:50.669106604 (49h54m26.708679724s) [upload=43849]: Done uploading segments... Sep 8 15:22:50.669125721 (49h54m26.708698841s) [upload=43849]: Upload finished. err: <nil> Duration: 9.300057656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X........X...] duration: 20.952422625s FAIL (2025-09-08 15:26:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:22:47.727033066 (49h54m23.766606187s) [upload=43849]: Starting upload Sep 8 15:22:47.727221938 (49h54m23.766795068s) [upload=43849]: Done starting upload Sep 8 15:22:47.727349863 (49h54m23.766922983s) [upload=43849]: Uploading segments... Sep 8 15:22:47.727494449 (49h54m23.76706757s) [upload=43849]: Got next segment. Inline: false Sep 8 15:22:47.727525218 (49h54m23.767098338s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:22:47.727529857 (49h54m23.767102977s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:22:47.993405316 (49h54m24.032978436s) [seg_pos={0 0}, upload=43849]: Begin upload segment... Sep 8 15:22:47.993452405 (49h54m24.033025525s) [seg_pos={0 0}, upload=43849]: Done begin upload segment. Sep 8 15:22:47.993477513 (49h54m24.033050633s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993483054 (49h54m24.033056174s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993861548 (49h54m24.033434678s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994021384 (49h54m24.033594514s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994387685 (49h54m24.033960815s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994688891 (49h54m24.034262011s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994751952 (49h54m24.034325072s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994922869 (49h54m24.034495989s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995228193 (49h54m24.034801313s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995337603 (49h54m24.034910723s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995487019 (49h54m24.035060139s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995661993 (49h54m24.035235113s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995782233 (49h54m24.035355353s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995935306 (49h54m24.035508426s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996023996 (49h54m24.035597116s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996149997 (49h54m24.035723117s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:48.253260102 (49h54m24.292833222s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57090->77.90.31.16:4735: write: connection refused; conn closed: op Sep 8 15:22:48.275865279 (49h54m24.315438399s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45466->77.90.31.16:4733: write: connection refused; conn closed: op Sep 8 15:22:49.850841260 (49h54m25.89041439s) [upload=43849]: Next returned nil segment Sep 8 15:22:49.850865206 (49h54m25.890438356s) [upload=43849]: Waiting for error group managing segments... Sep 8 15:22:50.175466458 (49h54m26.215039578s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.187812522 (49h54m26.227385642s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.195784016 (49h54m26.235357136s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.201113334 (49h54m26.240686454s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.209776070 (49h54m26.24934919s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.228795255 (49h54m26.268368375s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.232605095 (49h54m26.272178215s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.233700721 (49h54m26.273273831s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.290809673 (49h54m26.330382803s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.298656217 (49h54m26.338229337s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.307520770 (49h54m26.34709389s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.323901867 (49h54m26.363474987s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342940047 (49h54m26.382513197s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342987949 (49h54m26.382561069s) [seg_pos={0 0}, upload=43849]: Segment reached optimal threshold of 13 pieces. Sep 8 15:22:50.343193583 (49h54m26.382766703s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:22:50.343221546 (49h54m26.382794666s) [seg_pos={0 0}, upload=43849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:22:50.343287392 (49h54m26.382860512s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:22:50.343292141 (49h54m26.382865261s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:22:50.669106604 (49h54m26.708679724s) [upload=43849]: Done uploading segments... Sep 8 15:22:50.669125721 (49h54m26.708698841s) [upload=43849]: Upload finished. err: <nil> Duration: 20.952453494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X........X..] duration: 7.969811356s FAIL (2025-09-08 15:25:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:22:47.727033066 (49h54m23.766606187s) [upload=43849]: Starting upload Sep 8 15:22:47.727221938 (49h54m23.766795068s) [upload=43849]: Done starting upload Sep 8 15:22:47.727349863 (49h54m23.766922983s) [upload=43849]: Uploading segments... Sep 8 15:22:47.727494449 (49h54m23.76706757s) [upload=43849]: Got next segment. Inline: false Sep 8 15:22:47.727525218 (49h54m23.767098338s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:22:47.727529857 (49h54m23.767102977s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:22:47.993405316 (49h54m24.032978436s) [seg_pos={0 0}, upload=43849]: Begin upload segment... Sep 8 15:22:47.993452405 (49h54m24.033025525s) [seg_pos={0 0}, upload=43849]: Done begin upload segment. Sep 8 15:22:47.993477513 (49h54m24.033050633s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993483054 (49h54m24.033056174s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993861548 (49h54m24.033434678s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994021384 (49h54m24.033594514s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994387685 (49h54m24.033960815s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994688891 (49h54m24.034262011s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994751952 (49h54m24.034325072s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994922869 (49h54m24.034495989s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995228193 (49h54m24.034801313s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995337603 (49h54m24.034910723s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995487019 (49h54m24.035060139s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995661993 (49h54m24.035235113s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995782233 (49h54m24.035355353s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995935306 (49h54m24.035508426s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996023996 (49h54m24.035597116s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996149997 (49h54m24.035723117s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:48.253260102 (49h54m24.292833222s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57090->77.90.31.16:4735: write: connection refused; conn closed: op Sep 8 15:22:48.275865279 (49h54m24.315438399s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45466->77.90.31.16:4733: write: connection refused; conn closed: op Sep 8 15:22:49.850841260 (49h54m25.89041439s) [upload=43849]: Next returned nil segment Sep 8 15:22:49.850865206 (49h54m25.890438356s) [upload=43849]: Waiting for error group managing segments... Sep 8 15:22:50.175466458 (49h54m26.215039578s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.187812522 (49h54m26.227385642s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.195784016 (49h54m26.235357136s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.201113334 (49h54m26.240686454s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.209776070 (49h54m26.24934919s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.228795255 (49h54m26.268368375s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.232605095 (49h54m26.272178215s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.233700721 (49h54m26.273273831s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.290809673 (49h54m26.330382803s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.298656217 (49h54m26.338229337s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.307520770 (49h54m26.34709389s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.323901867 (49h54m26.363474987s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342940047 (49h54m26.382513197s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342987949 (49h54m26.382561069s) [seg_pos={0 0}, upload=43849]: Segment reached optimal threshold of 13 pieces. Sep 8 15:22:50.343193583 (49h54m26.382766703s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:22:50.343221546 (49h54m26.382794666s) [seg_pos={0 0}, upload=43849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:22:50.343287392 (49h54m26.382860512s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:22:50.343292141 (49h54m26.382865261s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:22:50.669106604 (49h54m26.708679724s) [upload=43849]: Done uploading segments... Sep 8 15:22:50.669125721 (49h54m26.708698841s) [upload=43849]: Upload finished. err: <nil> Duration: 7.969842205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X........X.] duration: 20.924107273s FAIL (2025-09-08 15:24:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:22:47.727033066 (49h54m23.766606187s) [upload=43849]: Starting upload Sep 8 15:22:47.727221938 (49h54m23.766795068s) [upload=43849]: Done starting upload Sep 8 15:22:47.727349863 (49h54m23.766922983s) [upload=43849]: Uploading segments... Sep 8 15:22:47.727494449 (49h54m23.76706757s) [upload=43849]: Got next segment. Inline: false Sep 8 15:22:47.727525218 (49h54m23.767098338s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:22:47.727529857 (49h54m23.767102977s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:22:47.993405316 (49h54m24.032978436s) [seg_pos={0 0}, upload=43849]: Begin upload segment... Sep 8 15:22:47.993452405 (49h54m24.033025525s) [seg_pos={0 0}, upload=43849]: Done begin upload segment. Sep 8 15:22:47.993477513 (49h54m24.033050633s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993483054 (49h54m24.033056174s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993861548 (49h54m24.033434678s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994021384 (49h54m24.033594514s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994387685 (49h54m24.033960815s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994688891 (49h54m24.034262011s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994751952 (49h54m24.034325072s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994922869 (49h54m24.034495989s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995228193 (49h54m24.034801313s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995337603 (49h54m24.034910723s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995487019 (49h54m24.035060139s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995661993 (49h54m24.035235113s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995782233 (49h54m24.035355353s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995935306 (49h54m24.035508426s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996023996 (49h54m24.035597116s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996149997 (49h54m24.035723117s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:48.253260102 (49h54m24.292833222s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57090->77.90.31.16:4735: write: connection refused; conn closed: op Sep 8 15:22:48.275865279 (49h54m24.315438399s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45466->77.90.31.16:4733: write: connection refused; conn closed: op Sep 8 15:22:49.850841260 (49h54m25.89041439s) [upload=43849]: Next returned nil segment Sep 8 15:22:49.850865206 (49h54m25.890438356s) [upload=43849]: Waiting for error group managing segments... Sep 8 15:22:50.175466458 (49h54m26.215039578s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.187812522 (49h54m26.227385642s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.195784016 (49h54m26.235357136s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.201113334 (49h54m26.240686454s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.209776070 (49h54m26.24934919s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.228795255 (49h54m26.268368375s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.232605095 (49h54m26.272178215s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.233700721 (49h54m26.273273831s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.290809673 (49h54m26.330382803s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.298656217 (49h54m26.338229337s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.307520770 (49h54m26.34709389s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.323901867 (49h54m26.363474987s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342940047 (49h54m26.382513197s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342987949 (49h54m26.382561069s) [seg_pos={0 0}, upload=43849]: Segment reached optimal threshold of 13 pieces. Sep 8 15:22:50.343193583 (49h54m26.382766703s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:22:50.343221546 (49h54m26.382794666s) [seg_pos={0 0}, upload=43849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:22:50.343287392 (49h54m26.382860512s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:22:50.343292141 (49h54m26.382865261s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:22:50.669106604 (49h54m26.708679724s) [upload=43849]: Done uploading segments... Sep 8 15:22:50.669125721 (49h54m26.708698841s) [upload=43849]: Upload finished. err: <nil> Duration: 20.924134556s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X........X] duration: 30.002033996s FAIL (2025-09-08 15:23:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:22:47.727033066 (49h54m23.766606187s) [upload=43849]: Starting upload Sep 8 15:22:47.727221938 (49h54m23.766795068s) [upload=43849]: Done starting upload Sep 8 15:22:47.727349863 (49h54m23.766922983s) [upload=43849]: Uploading segments... Sep 8 15:22:47.727494449 (49h54m23.76706757s) [upload=43849]: Got next segment. Inline: false Sep 8 15:22:47.727525218 (49h54m23.767098338s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:22:47.727529857 (49h54m23.767102977s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:22:47.993405316 (49h54m24.032978436s) [seg_pos={0 0}, upload=43849]: Begin upload segment... Sep 8 15:22:47.993452405 (49h54m24.033025525s) [seg_pos={0 0}, upload=43849]: Done begin upload segment. Sep 8 15:22:47.993477513 (49h54m24.033050633s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993483054 (49h54m24.033056174s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.993861548 (49h54m24.033434678s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994021384 (49h54m24.033594514s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994387685 (49h54m24.033960815s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994688891 (49h54m24.034262011s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994751952 (49h54m24.034325072s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.994922869 (49h54m24.034495989s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995228193 (49h54m24.034801313s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995337603 (49h54m24.034910723s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995487019 (49h54m24.035060139s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995661993 (49h54m24.035235113s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995782233 (49h54m24.035355353s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.995935306 (49h54m24.035508426s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996023996 (49h54m24.035597116s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:47.996149997 (49h54m24.035723117s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Uploading piece... Sep 8 15:22:48.253260102 (49h54m24.292833222s) [noise=true , address=77.90.31.16:4735 , piece_id=WDJ6BGNA27FZXK66AG5IO345KW5ELKZBASNXST7ZXGFKPPPLPPFA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57090->77.90.31.16:4735: write: connection refused; conn closed: op Sep 8 15:22:48.275865279 (49h54m24.315438399s) [noise=true , address=77.90.31.16:4733 , piece_id=APPBHNWJCA4TQVLTVQNZBA4YRQPF2JXQKZFL3OUQZ27ULNJUKUSA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45466->77.90.31.16:4733: write: connection refused; conn closed: op Sep 8 15:22:49.850841260 (49h54m25.89041439s) [upload=43849]: Next returned nil segment Sep 8 15:22:49.850865206 (49h54m25.890438356s) [upload=43849]: Waiting for error group managing segments... Sep 8 15:22:50.175466458 (49h54m26.215039578s) [noise=true , address=45.91.116.5:4297 , piece_id=DMQT2HPPI3SADTW3RJZNKVVY463S6GOOFSGR7XEJBH2A5ALFY3CQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.187812522 (49h54m26.227385642s) [noise=true , address=45.91.116.13:4153 , piece_id=O2NISMJNE2OEWJV6YKCI5TQJ55QSR4STWNCSSNUWWVXDOJMNBPDQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.195784016 (49h54m26.235357136s) [noise=true , address=45.91.116.10:4083 , piece_id=WN44PZQOX4F4ZJEJQ7CAA6YX7BUGV6DKB4QTDNZVN7T7YUFQFUNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.201113334 (49h54m26.240686454s) [noise=true , address=45.91.116.10:4081 , piece_id=Y25Y5ZPILIJTEXHSHTMUMAPLW6N5DPOROFMTHYKTB4YJIVSQX64Q, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.209776070 (49h54m26.24934919s) [noise=true , address=77.90.31.19:4843 , piece_id=U7HW6Y52G7AT2HPGIW6GYARQWV6747JDY45GE4OI4HPOAD34AVSQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.228795255 (49h54m26.268368375s) [noise=true , address=77.90.31.12:4592 , piece_id=S4HCZI7BSSQRVOLTJNUXT6RFPIKMEX5UIQRJFVZON7OFCX55FGQQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.232605095 (49h54m26.272178215s) [noise=true , address=45.91.116.11:4119 , piece_id=XWTVBOOXY7XV75YNARM7WF6LXJQI2BKO2KXZKGUGXWO5IBE32MCA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.233700721 (49h54m26.273273831s) [noise=true , address=77.90.31.18:4806 , piece_id=DU27YLFXAQJWGRZLUE7MG4A4J6FTUMIZ6GYHGMEK2WOJJTDLUDJA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.290809673 (49h54m26.330382803s) [noise=true , address=45.91.116.12:4226 , piece_id=KC4NPYH4LQ5IMDSEGKBCWXBGE5PI7NEEBHJ2JEQVAT3LP4B3Y3YQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.298656217 (49h54m26.338229337s) [noise=true , address=145.79.227.18:4007 , piece_id=4D4RLVS27MENSS6PG336FURM3R2C4YPLK56FSWCLYNP2SN2SRLNQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.307520770 (49h54m26.34709389s) [noise=true , address=45.91.116.6:4975 , piece_id=UUVHPMRSCE4LWF6IA2ALVRVO4E7XTRDOV5V5OLB4OPFZWOMR2KTQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.323901867 (49h54m26.363474987s) [noise=true , address=77.90.31.17:4771 , piece_id=2A3C24MCIXBOUFVAV43WZDVRMXXKVOBWR3K5UXY2B7NU3REBM5PA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342940047 (49h54m26.382513197s) [noise=true , address=45.91.116.9:4263 , piece_id=PKEXYGMPNPJ44SIQEQSGZJYML3H3JXC5LW6OAAZSWFULAKKYSSPA, seg_pos={0 0}, upload=43849]: Done uploading piece. err: <nil> Sep 8 15:22:50.342987949 (49h54m26.382561069s) [seg_pos={0 0}, upload=43849]: Segment reached optimal threshold of 13 pieces. Sep 8 15:22:50.343193583 (49h54m26.382766703s) [noise=true , address=45.91.116.6:4974 , piece_id=IZGJUNRUEGR57SR5ETRYBV6BYNJKQMO46KCC3QXS5ZHIJB4GONVQ, seg_pos={0 0}, upload=43849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:22:50.343221546 (49h54m26.382794666s) [seg_pos={0 0}, upload=43849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:22:50.343287392 (49h54m26.382860512s) [upload=43849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:22:50.343292141 (49h54m26.382865261s) [upload=43849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:22:50.669106604 (49h54m26.708679724s) [upload=43849]: Done uploading segments... Sep 8 15:22:50.669125721 (49h54m26.708698841s) [upload=43849]: Upload finished. err: <nil> Duration: 30.00206236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X........] duration: 13.51488934s FAIL (2025-09-08 15:22:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 13.514918518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X.......] duration: 15.540614399s FAIL (2025-09-08 15:21:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 15.540638113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X......] duration: 7.262703469s FAIL (2025-09-08 15:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 7.262731001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X.....] duration: 10.305294218s FAIL (2025-09-08 15:20:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 10.305316091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X....] duration: 12.298347002s FAIL (2025-09-08 15:19:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 12.298367561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X...] duration: 17.339769264s FAIL (2025-09-08 15:17:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 17.339807858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X..] duration: 11.787109344s FAIL (2025-09-08 15:17:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 11.787133059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X.] duration: 7.242284895s FAIL (2025-09-08 15:16:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 7.242322507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.X] duration: 30.001606259s FAIL (2025-09-08 15:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:14:59.767094280 (49h46m35.8066674s) [upload=43736]: Starting upload Sep 8 15:14:59.767163181 (49h46m35.806736301s) [upload=43736]: Done starting upload Sep 8 15:14:59.767239397 (49h46m35.806812517s) [upload=43736]: Uploading segments... Sep 8 15:14:59.767462915 (49h46m35.807036035s) [upload=43736]: Got next segment. Inline: false Sep 8 15:14:59.767487552 (49h46m35.807060672s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:14:59.767491629 (49h46m35.80706475s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:15:00.301427806 (49h46m36.341000936s) [seg_pos={0 0}, upload=43736]: Begin upload segment... Sep 8 15:15:00.301494404 (49h46m36.341067524s) [seg_pos={0 0}, upload=43736]: Done begin upload segment. Sep 8 15:15:00.301518009 (49h46m36.341091129s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301541805 (49h46m36.341114925s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301925449 (49h46m36.341498569s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.301934917 (49h46m36.341508037s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302213139 (49h46m36.341786259s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302221204 (49h46m36.341794324s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302591453 (49h46m36.342164573s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.302713135 (49h46m36.342286265s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303219214 (49h46m36.342792334s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303498067 (49h46m36.343071188s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.303830894 (49h46m36.343404014s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304143533 (49h46m36.343716653s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304666423 (49h46m36.344239543s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.304971898 (49h46m36.344545018s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305424043 (49h46m36.344997163s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:00.305764575 (49h46m36.345337695s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Uploading piece... Sep 8 15:15:02.150298111 (49h46m38.189871242s) [upload=43736]: Next returned nil segment Sep 8 15:15:02.150323610 (49h46m38.18989674s) [upload=43736]: Waiting for error group managing segments... Sep 8 15:15:02.463574473 (49h46m38.503147603s) [noise=true , address=45.91.116.6:4975 , piece_id=3HJK3A3CRLBEWS4RAWD5TC2I4LPQBRPJ2L7XYU7MHHDGW3D23BSQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.470198909 (49h46m38.509772029s) [noise=true , address=45.91.116.4:4937 , piece_id=MOOUWMZK5GV6XZBESEXW3MGIC4TVKO2P65AHAIZKA3TOLX5FVZMA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.479724888 (49h46m38.519298018s) [noise=true , address=45.91.116.6:4973 , piece_id=5LHL4NNQBN6ZLA3B72J27LEHBFWGW7Q2HSBESVZUGIM4K7CS2NCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.486323714 (49h46m38.525896834s) [noise=true , address=45.91.116.13:4153 , piece_id=W7SLH64GFV4MPRY4HJC34HZFGRCNB54GQIDN7WT7LODHAM3BU3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.529859313 (49h46m38.569432433s) [noise=true , address=45.91.116.14:4189 , piece_id=U26GFUKJ2RQSLMOZC34LM2RZ4EJ7ATXFQJGBRL2MG3D4ZE6IT7YQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.541305385 (49h46m38.580878505s) [noise=true , address=45.91.116.9:4261 , piece_id=6NDBW6ZTHYXGFSIPGQJJW7JYC3DNZJ4VNKLFQJTHZONNR23YVHCQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.568827500 (49h46m38.60840062s) [noise=true , address=45.91.116.5:4298 , piece_id=34NRZWEKT5PSIDNSRH5YHJME7IUWVJQ4BSIAYVM5KHCUA3577PGA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.579744481 (49h46m38.619317601s) [noise=true , address=45.91.116.4:4940 , piece_id=U33263NUDQUBE2VWDZPHAYMH3KGRAYC5JIA54ZBZPIXNEM7HSSZA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.610394731 (49h46m38.649967861s) [noise=true , address=45.91.116.8:4045 , piece_id=MPCLNRI56JEF25UHDKZJKF36YV7R3VPYKHL2MBSLJEDAFXM2AWPA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.680339635 (49h46m38.719912765s) [noise=true , address=77.90.31.19:4843 , piece_id=URUTFLE4FBYUTNAAE6FY67SKR4ITO5FELZ7JSPURWLZ6SSJAO5WQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:02.851319935 (49h46m38.890893065s) [noise=true , address=145.79.227.6:4258 , piece_id=QP565OVUT7FP3VH725SFKTIDTUFDQTDBIYDPP6UD3PYHC36IEQTQ, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.064871077 (49h46m39.104444207s) [noise=true , address=77.90.31.18:4807 , piece_id=YRK5WBN44RAGVYAW3E7L5RXNOHNXEY4QPLDREHTPYSMJC4POL3SA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215313235 (49h46m39.254886365s) [noise=true , address=145.79.227.4:4186 , piece_id=PXRM6FQMDN4AEMJAU3BTSUPCPFJIP5ZUAYB22TE36E7YY27NIM3A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: <nil> Sep 8 15:15:03.215355697 (49h46m39.254928827s) [seg_pos={0 0}, upload=43736]: Segment reached optimal threshold of 13 pieces. Sep 8 15:15:03.215529419 (49h46m39.255102549s) [noise=true , address=77.90.31.12:4592 , piece_id=UWONDONPNJGMUDU3XQU7AVYHPXJ3OIHTNSIYJIBT754YISOT2O2A, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215630382 (49h46m39.255203502s) [noise=true , address=45.91.116.13:4155 , piece_id=Z2USRM7F7QAQVILTVCG6EQJD5UN3DEU3E43YCCND6MAQI5JWJUJA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 8 15:15:03.215753408 (49h46m39.255326538s) [noise=true , address=45.91.116.7:4011 , piece_id=UN6L6ZJBDDXWRWGNUSZQNTBSWQYC3BICH34GXUSY7IN3TENBWZCA, seg_pos={0 0}, upload=43736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33826->45.91.116.7:4011: use of closed network connection Sep 8 15:15:03.215806018 (49h46m39.255379148s) [seg_pos={0 0}, upload=43736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:15:03.215919616 (49h46m39.255492746s) [upload=43736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:15:03.215927591 (49h46m39.255500721s) [upload=43736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:15:03.577692919 (49h46m39.617266039s) [upload=43736]: Done uploading segments... Sep 8 15:15:03.577714570 (49h46m39.61728769s) [upload=43736]: Upload finished. err: <nil> Duration: 30.001629484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X.] duration: 10.165961321s FAIL (2025-09-08 15:13:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:12:48.571973093 (49h44m24.611546213s) [upload=43705]: Starting upload Sep 8 15:12:48.572087131 (49h44m24.611660282s) [upload=43705]: Done starting upload Sep 8 15:12:48.572154420 (49h44m24.61172757s) [upload=43705]: Uploading segments... Sep 8 15:12:48.572433474 (49h44m24.612006584s) [upload=43705]: Got next segment. Inline: false Sep 8 15:12:48.572490153 (49h44m24.612063273s) [upload=43705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:12:48.572494551 (49h44m24.612067671s) [upload=43705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:12:49.087849871 (49h44m25.127422992s) [seg_pos={0 0}, upload=43705]: Begin upload segment... Sep 8 15:12:49.087902823 (49h44m25.127475933s) [seg_pos={0 0}, upload=43705]: Done begin upload segment. Sep 8 15:12:49.087924544 (49h44m25.127497664s) [noise=true , address=145.79.227.18:4005 , piece_id=N55LRIHAFPAZRKOQQPXMWPVPBIYXZRNIKMEYROGA5GJ5CMQ5JEFA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.088515345 (49h44m25.128088465s) [noise=true , address=45.91.116.8:4048 , piece_id=WZA6JHOWAIUZ6GRWN5WR74CDCOSO2TH72KUIG22WMRWZTS36TH2A, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.088838333 (49h44m25.128411453s) [noise=true , address=45.91.116.4:4938 , piece_id=BBUXT52RBOMEKI4P2RBOCPAXN3OYY2RTCZIKMEY53IZNNLUO3BBQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.089174065 (49h44m25.128747185s) [noise=true , address=77.90.31.18:4806 , piece_id=GPRROSLES42PZDOHYTBV7OIGEOGEH62OMLBBSELT73FVXNKWDNJA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.089437720 (49h44m25.12901084s) [noise=true , address=45.91.116.6:4973 , piece_id=SW6USTSFMOQTKPYXPUWZ2V55E4QVKPO52U4YBEXDWIYQLHN2KFPQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.089777309 (49h44m25.129350419s) [noise=true , address=77.90.31.17:4770 , piece_id=SNHUD557NBY5VJBAAZWZVFOVA3Y66ND254C7WAYTTOUVHBZP7L3Q, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.090939243 (49h44m25.130512353s) [noise=true , address=77.90.31.17:4771 , piece_id=YGPVQU3SEEPAUWZ3O7ZOU2XBGLUEBT4KSIJTQKRXDE4KQJD3U5QA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.091249166 (49h44m25.130822276s) [noise=true , address=45.91.116.4:4940 , piece_id=CFI62QFJLHQ7FYCWOPBXVQQN2PPHDJJK23SGNGAKNM5ZOTD65LEQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.091514644 (49h44m25.131087764s) [noise=true , address=45.91.116.7:4010 , piece_id=MROQCJHGP2NFATWMLFAFARDLNCWF74AN3GNHF64DRYBFTRAUF6EA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.091788508 (49h44m25.131361628s) [noise=true , address=145.79.227.14:4839 , piece_id=7R64W7S2BVMUKPBDEIB3BH4U7EN6HDYVBRKB53PEGP3QPCSXLUZQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092043826 (49h44m25.131616936s) [noise=true , address=77.90.31.12:4589 , piece_id=AXTZS7PUVZ24ULR5S6HHJJVNVGBUKOBSVKN42DWIX22W4E32ZU6A, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092289256 (49h44m25.131862376s) [noise=true , address=45.91.116.5:4298 , piece_id=NPBIPAY5UFJNDIE3CXTRXPIS6LDFXFSTYFUS46YI64T3YOINR5JA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092521611 (49h44m25.132094731s) [noise=true , address=77.90.31.10:4519 , piece_id=ZFX6GIRSE53TJBR27XYHK5SW2NYHLEM5C5KWQYTQJFG4JB2S622Q, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092751340 (49h44m25.13232446s) [noise=true , address=145.79.227.11:4730 , piece_id=T2CLHRW2YKJL224OYBKEKOQSIRVRPUINFETUDWJEBBWW5OIU42JA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092906477 (49h44m25.132479597s) [noise=true , address=45.91.116.13:4155 , piece_id=EGZ3QLZHVVOXFM3GZQCADD3JVF4POF7IOUNVZXVY4KKXHFQ3YW2Q, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.093252920 (49h44m25.13282605s) [noise=true , address=45.91.116.4:4939 , piece_id=WWXMXMMHC6QLKFHJHC2UBIM2H6NNE2KBAT7UFE3GH667774DCIKQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:51.073380061 (49h44m27.112953181s) [upload=43705]: Next returned nil segment Sep 8 15:12:51.073395962 (49h44m27.112969082s) [upload=43705]: Waiting for error group managing segments... Sep 8 15:12:51.448880091 (49h44m27.488453211s) [noise=true , address=45.91.116.6:4973 , piece_id=SW6USTSFMOQTKPYXPUWZ2V55E4QVKPO52U4YBEXDWIYQLHN2KFPQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.503517212 (49h44m27.543090332s) [noise=true , address=77.90.31.10:4519 , piece_id=ZFX6GIRSE53TJBR27XYHK5SW2NYHLEM5C5KWQYTQJFG4JB2S622Q, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.550447646 (49h44m27.590020766s) [noise=true , address=77.90.31.17:4770 , piece_id=SNHUD557NBY5VJBAAZWZVFOVA3Y66ND254C7WAYTTOUVHBZP7L3Q, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.592758780 (49h44m27.63233189s) [noise=true , address=45.91.116.7:4010 , piece_id=MROQCJHGP2NFATWMLFAFARDLNCWF74AN3GNHF64DRYBFTRAUF6EA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.632457105 (49h44m27.672030215s) [noise=true , address=77.90.31.12:4589 , piece_id=AXTZS7PUVZ24ULR5S6HHJJVNVGBUKOBSVKN42DWIX22W4E32ZU6A, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.661660759 (49h44m27.701233879s) [noise=true , address=77.90.31.18:4806 , piece_id=GPRROSLES42PZDOHYTBV7OIGEOGEH62OMLBBSELT73FVXNKWDNJA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.661990219 (49h44m27.701563329s) [noise=true , address=45.91.116.4:4938 , piece_id=BBUXT52RBOMEKI4P2RBOCPAXN3OYY2RTCZIKMEY53IZNNLUO3BBQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.716348897 (49h44m27.755922017s) [noise=true , address=45.91.116.5:4298 , piece_id=NPBIPAY5UFJNDIE3CXTRXPIS6LDFXFSTYFUS46YI64T3YOINR5JA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.721992496 (49h44m27.761565606s) [noise=true , address=45.91.116.8:4048 , piece_id=WZA6JHOWAIUZ6GRWN5WR74CDCOSO2TH72KUIG22WMRWZTS36TH2A, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.806051528 (49h44m27.845624648s) [noise=true , address=45.91.116.13:4155 , piece_id=EGZ3QLZHVVOXFM3GZQCADD3JVF4POF7IOUNVZXVY4KKXHFQ3YW2Q, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.875933891 (49h44m27.915507011s) [noise=true , address=45.91.116.4:4939 , piece_id=WWXMXMMHC6QLKFHJHC2UBIM2H6NNE2KBAT7UFE3GH667774DCIKQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.878402105 (49h44m27.917975225s) [noise=true , address=77.90.31.17:4771 , piece_id=YGPVQU3SEEPAUWZ3O7ZOU2XBGLUEBT4KSIJTQKRXDE4KQJD3U5QA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.908413133 (49h44m27.947986253s) [noise=true , address=45.91.116.4:4940 , piece_id=CFI62QFJLHQ7FYCWOPBXVQQN2PPHDJJK23SGNGAKNM5ZOTD65LEQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.908432158 (49h44m27.948005278s) [seg_pos={0 0}, upload=43705]: Segment reached optimal threshold of 13 pieces. Sep 8 15:12:51.908633685 (49h44m27.948206855s) [noise=true , address=145.79.227.14:4839 , piece_id=7R64W7S2BVMUKPBDEIB3BH4U7EN6HDYVBRKB53PEGP3QPCSXLUZQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Sep 8 15:12:51.908710812 (49h44m27.948283973s) [noise=true , address=145.79.227.18:4005 , piece_id=N55LRIHAFPAZRKOQQPXMWPVPBIYXZRNIKMEYROGA5GJ5CMQ5JEFA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:12:51.908798570 (49h44m27.94837168s) [noise=true , address=145.79.227.11:4730 , piece_id=T2CLHRW2YKJL224OYBKEKOQSIRVRPUINFETUDWJEBBWW5OIU42JA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:46678->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46678->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46678->145.79.227.11:4730: use of closed network connection Sep 8 15:12:51.908812166 (49h44m27.948385286s) [seg_pos={0 0}, upload=43705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:12:51.908907028 (49h44m27.948480208s) [upload=43705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:12:51.908915303 (49h44m27.948488463s) [upload=43705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:12:52.253386127 (49h44m28.292959257s) [upload=43705]: Done uploading segments... Sep 8 15:12:52.253406035 (49h44m28.292979156s) [upload=43705]: Upload finished. err: <nil> Duration: 10.165989344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......X] duration: 30.002340845s FAIL (2025-09-08 15:13:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:12:48.571973093 (49h44m24.611546213s) [upload=43705]: Starting upload Sep 8 15:12:48.572087131 (49h44m24.611660282s) [upload=43705]: Done starting upload Sep 8 15:12:48.572154420 (49h44m24.61172757s) [upload=43705]: Uploading segments... Sep 8 15:12:48.572433474 (49h44m24.612006584s) [upload=43705]: Got next segment. Inline: false Sep 8 15:12:48.572490153 (49h44m24.612063273s) [upload=43705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:12:48.572494551 (49h44m24.612067671s) [upload=43705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:12:49.087849871 (49h44m25.127422992s) [seg_pos={0 0}, upload=43705]: Begin upload segment... Sep 8 15:12:49.087902823 (49h44m25.127475933s) [seg_pos={0 0}, upload=43705]: Done begin upload segment. Sep 8 15:12:49.087924544 (49h44m25.127497664s) [noise=true , address=145.79.227.18:4005 , piece_id=N55LRIHAFPAZRKOQQPXMWPVPBIYXZRNIKMEYROGA5GJ5CMQ5JEFA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.088515345 (49h44m25.128088465s) [noise=true , address=45.91.116.8:4048 , piece_id=WZA6JHOWAIUZ6GRWN5WR74CDCOSO2TH72KUIG22WMRWZTS36TH2A, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.088838333 (49h44m25.128411453s) [noise=true , address=45.91.116.4:4938 , piece_id=BBUXT52RBOMEKI4P2RBOCPAXN3OYY2RTCZIKMEY53IZNNLUO3BBQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.089174065 (49h44m25.128747185s) [noise=true , address=77.90.31.18:4806 , piece_id=GPRROSLES42PZDOHYTBV7OIGEOGEH62OMLBBSELT73FVXNKWDNJA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.089437720 (49h44m25.12901084s) [noise=true , address=45.91.116.6:4973 , piece_id=SW6USTSFMOQTKPYXPUWZ2V55E4QVKPO52U4YBEXDWIYQLHN2KFPQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.089777309 (49h44m25.129350419s) [noise=true , address=77.90.31.17:4770 , piece_id=SNHUD557NBY5VJBAAZWZVFOVA3Y66ND254C7WAYTTOUVHBZP7L3Q, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.090939243 (49h44m25.130512353s) [noise=true , address=77.90.31.17:4771 , piece_id=YGPVQU3SEEPAUWZ3O7ZOU2XBGLUEBT4KSIJTQKRXDE4KQJD3U5QA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.091249166 (49h44m25.130822276s) [noise=true , address=45.91.116.4:4940 , piece_id=CFI62QFJLHQ7FYCWOPBXVQQN2PPHDJJK23SGNGAKNM5ZOTD65LEQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.091514644 (49h44m25.131087764s) [noise=true , address=45.91.116.7:4010 , piece_id=MROQCJHGP2NFATWMLFAFARDLNCWF74AN3GNHF64DRYBFTRAUF6EA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.091788508 (49h44m25.131361628s) [noise=true , address=145.79.227.14:4839 , piece_id=7R64W7S2BVMUKPBDEIB3BH4U7EN6HDYVBRKB53PEGP3QPCSXLUZQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092043826 (49h44m25.131616936s) [noise=true , address=77.90.31.12:4589 , piece_id=AXTZS7PUVZ24ULR5S6HHJJVNVGBUKOBSVKN42DWIX22W4E32ZU6A, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092289256 (49h44m25.131862376s) [noise=true , address=45.91.116.5:4298 , piece_id=NPBIPAY5UFJNDIE3CXTRXPIS6LDFXFSTYFUS46YI64T3YOINR5JA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092521611 (49h44m25.132094731s) [noise=true , address=77.90.31.10:4519 , piece_id=ZFX6GIRSE53TJBR27XYHK5SW2NYHLEM5C5KWQYTQJFG4JB2S622Q, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092751340 (49h44m25.13232446s) [noise=true , address=145.79.227.11:4730 , piece_id=T2CLHRW2YKJL224OYBKEKOQSIRVRPUINFETUDWJEBBWW5OIU42JA, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.092906477 (49h44m25.132479597s) [noise=true , address=45.91.116.13:4155 , piece_id=EGZ3QLZHVVOXFM3GZQCADD3JVF4POF7IOUNVZXVY4KKXHFQ3YW2Q, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:49.093252920 (49h44m25.13282605s) [noise=true , address=45.91.116.4:4939 , piece_id=WWXMXMMHC6QLKFHJHC2UBIM2H6NNE2KBAT7UFE3GH667774DCIKQ, seg_pos={0 0}, upload=43705]: Uploading piece... Sep 8 15:12:51.073380061 (49h44m27.112953181s) [upload=43705]: Next returned nil segment Sep 8 15:12:51.073395962 (49h44m27.112969082s) [upload=43705]: Waiting for error group managing segments... Sep 8 15:12:51.448880091 (49h44m27.488453211s) [noise=true , address=45.91.116.6:4973 , piece_id=SW6USTSFMOQTKPYXPUWZ2V55E4QVKPO52U4YBEXDWIYQLHN2KFPQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.503517212 (49h44m27.543090332s) [noise=true , address=77.90.31.10:4519 , piece_id=ZFX6GIRSE53TJBR27XYHK5SW2NYHLEM5C5KWQYTQJFG4JB2S622Q, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.550447646 (49h44m27.590020766s) [noise=true , address=77.90.31.17:4770 , piece_id=SNHUD557NBY5VJBAAZWZVFOVA3Y66ND254C7WAYTTOUVHBZP7L3Q, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.592758780 (49h44m27.63233189s) [noise=true , address=45.91.116.7:4010 , piece_id=MROQCJHGP2NFATWMLFAFARDLNCWF74AN3GNHF64DRYBFTRAUF6EA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.632457105 (49h44m27.672030215s) [noise=true , address=77.90.31.12:4589 , piece_id=AXTZS7PUVZ24ULR5S6HHJJVNVGBUKOBSVKN42DWIX22W4E32ZU6A, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.661660759 (49h44m27.701233879s) [noise=true , address=77.90.31.18:4806 , piece_id=GPRROSLES42PZDOHYTBV7OIGEOGEH62OMLBBSELT73FVXNKWDNJA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.661990219 (49h44m27.701563329s) [noise=true , address=45.91.116.4:4938 , piece_id=BBUXT52RBOMEKI4P2RBOCPAXN3OYY2RTCZIKMEY53IZNNLUO3BBQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.716348897 (49h44m27.755922017s) [noise=true , address=45.91.116.5:4298 , piece_id=NPBIPAY5UFJNDIE3CXTRXPIS6LDFXFSTYFUS46YI64T3YOINR5JA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.721992496 (49h44m27.761565606s) [noise=true , address=45.91.116.8:4048 , piece_id=WZA6JHOWAIUZ6GRWN5WR74CDCOSO2TH72KUIG22WMRWZTS36TH2A, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.806051528 (49h44m27.845624648s) [noise=true , address=45.91.116.13:4155 , piece_id=EGZ3QLZHVVOXFM3GZQCADD3JVF4POF7IOUNVZXVY4KKXHFQ3YW2Q, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.875933891 (49h44m27.915507011s) [noise=true , address=45.91.116.4:4939 , piece_id=WWXMXMMHC6QLKFHJHC2UBIM2H6NNE2KBAT7UFE3GH667774DCIKQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.878402105 (49h44m27.917975225s) [noise=true , address=77.90.31.17:4771 , piece_id=YGPVQU3SEEPAUWZ3O7ZOU2XBGLUEBT4KSIJTQKRXDE4KQJD3U5QA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.908413133 (49h44m27.947986253s) [noise=true , address=45.91.116.4:4940 , piece_id=CFI62QFJLHQ7FYCWOPBXVQQN2PPHDJJK23SGNGAKNM5ZOTD65LEQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: <nil> Sep 8 15:12:51.908432158 (49h44m27.948005278s) [seg_pos={0 0}, upload=43705]: Segment reached optimal threshold of 13 pieces. Sep 8 15:12:51.908633685 (49h44m27.948206855s) [noise=true , address=145.79.227.14:4839 , piece_id=7R64W7S2BVMUKPBDEIB3BH4U7EN6HDYVBRKB53PEGP3QPCSXLUZQ, seg_pos={0 0}, upload=43705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Sep 8 15:12:51.908710812 (49h44m27.948283973s) [noise=true , address=145.79.227.18:4005 , piece_id=N55LRIHAFPAZRKOQQPXMWPVPBIYXZRNIKMEYROGA5GJ5CMQ5JEFA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: expected piece hash; context canceled; EOF Sep 8 15:12:51.908798570 (49h44m27.94837168s) [noise=true , address=145.79.227.11:4730 , piece_id=T2CLHRW2YKJL224OYBKEKOQSIRVRPUINFETUDWJEBBWW5OIU42JA, seg_pos={0 0}, upload=43705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:46678->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46678->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46678->145.79.227.11:4730: use of closed network connection Sep 8 15:12:51.908812166 (49h44m27.948385286s) [seg_pos={0 0}, upload=43705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:12:51.908907028 (49h44m27.948480208s) [upload=43705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:12:51.908915303 (49h44m27.948488463s) [upload=43705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:12:52.253386127 (49h44m28.292959257s) [upload=43705]: Done uploading segments... Sep 8 15:12:52.253406035 (49h44m28.292979156s) [upload=43705]: Upload finished. err: <nil> Duration: 30.002364221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX......] duration: 7.356847807s FAIL (2025-09-08 15:12:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:07:35.933429542 (49h39m11.973002662s) [upload=43630]: Starting upload Sep 8 15:07:35.933522249 (49h39m11.973095369s) [upload=43630]: Done starting upload Sep 8 15:07:35.933619736 (49h39m11.973192856s) [upload=43630]: Uploading segments... Sep 8 15:07:35.933815170 (49h39m11.97338834s) [upload=43630]: Got next segment. Inline: false Sep 8 15:07:35.933860427 (49h39m11.973433537s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:07:35.933873071 (49h39m11.973446191s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:07:36.438715616 (49h39m12.478288736s) [seg_pos={0 0}, upload=43630]: Begin upload segment... Sep 8 15:07:36.438762556 (49h39m12.478335677s) [seg_pos={0 0}, upload=43630]: Done begin upload segment. Sep 8 15:07:36.438787144 (49h39m12.478360264s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.438837350 (49h39m12.47841047s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439289214 (49h39m12.478862334s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439431577 (49h39m12.479004697s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439568549 (49h39m12.479141669s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439761128 (49h39m12.479334248s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439880967 (49h39m12.479454087s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440178376 (49h39m12.479751496s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440239102 (49h39m12.479812222s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440752444 (49h39m12.480325584s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440898734 (49h39m12.480471884s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441275796 (49h39m12.480848916s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441730656 (49h39m12.481303786s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.442853735 (49h39m12.482426865s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443494030 (49h39m12.48306716s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443836766 (49h39m12.483409886s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:38.304825086 (49h39m14.344398206s) [upload=43630]: Next returned nil segment Sep 8 15:07:38.304843241 (49h39m14.344416361s) [upload=43630]: Waiting for error group managing segments... Sep 8 15:07:38.645664021 (49h39m14.685237141s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.650928124 (49h39m14.690501234s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.655750381 (49h39m14.695323501s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.669759469 (49h39m14.709332589s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.672660270 (49h39m14.71223339s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.711041505 (49h39m14.750614625s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.727027635 (49h39m14.766600755s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.792035711 (49h39m14.831608831s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.898063071 (49h39m14.937636192s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.008776007 (49h39m15.048349137s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.100047661 (49h39m15.139620781s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.241123978 (49h39m15.280697128s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484182555 (49h39m15.523755675s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484207583 (49h39m15.523780703s) [seg_pos={0 0}, upload=43630]: Segment reached optimal threshold of 13 pieces. Sep 8 15:07:39.484317854 (49h39m15.523890964s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484413617 (49h39m15.523986727s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484477289 (49h39m15.524050409s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection Sep 8 15:07:39.484496956 (49h39m15.524070076s) [seg_pos={0 0}, upload=43630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:07:39.484556120 (49h39m15.52412924s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:07:39.484560298 (49h39m15.524133418s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:07:39.799359894 (49h39m15.838933014s) [upload=43630]: Done uploading segments... Sep 8 15:07:39.799379402 (49h39m15.838952522s) [upload=43630]: Upload finished. err: <nil> Duration: 7.356891209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX.....] duration: 10.523027353s FAIL (2025-09-08 15:11:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:07:35.933429542 (49h39m11.973002662s) [upload=43630]: Starting upload Sep 8 15:07:35.933522249 (49h39m11.973095369s) [upload=43630]: Done starting upload Sep 8 15:07:35.933619736 (49h39m11.973192856s) [upload=43630]: Uploading segments... Sep 8 15:07:35.933815170 (49h39m11.97338834s) [upload=43630]: Got next segment. Inline: false Sep 8 15:07:35.933860427 (49h39m11.973433537s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:07:35.933873071 (49h39m11.973446191s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:07:36.438715616 (49h39m12.478288736s) [seg_pos={0 0}, upload=43630]: Begin upload segment... Sep 8 15:07:36.438762556 (49h39m12.478335677s) [seg_pos={0 0}, upload=43630]: Done begin upload segment. Sep 8 15:07:36.438787144 (49h39m12.478360264s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.438837350 (49h39m12.47841047s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439289214 (49h39m12.478862334s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439431577 (49h39m12.479004697s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439568549 (49h39m12.479141669s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439761128 (49h39m12.479334248s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439880967 (49h39m12.479454087s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440178376 (49h39m12.479751496s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440239102 (49h39m12.479812222s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440752444 (49h39m12.480325584s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440898734 (49h39m12.480471884s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441275796 (49h39m12.480848916s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441730656 (49h39m12.481303786s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.442853735 (49h39m12.482426865s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443494030 (49h39m12.48306716s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443836766 (49h39m12.483409886s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:38.304825086 (49h39m14.344398206s) [upload=43630]: Next returned nil segment Sep 8 15:07:38.304843241 (49h39m14.344416361s) [upload=43630]: Waiting for error group managing segments... Sep 8 15:07:38.645664021 (49h39m14.685237141s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.650928124 (49h39m14.690501234s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.655750381 (49h39m14.695323501s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.669759469 (49h39m14.709332589s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.672660270 (49h39m14.71223339s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.711041505 (49h39m14.750614625s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.727027635 (49h39m14.766600755s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.792035711 (49h39m14.831608831s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.898063071 (49h39m14.937636192s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.008776007 (49h39m15.048349137s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.100047661 (49h39m15.139620781s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.241123978 (49h39m15.280697128s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484182555 (49h39m15.523755675s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484207583 (49h39m15.523780703s) [seg_pos={0 0}, upload=43630]: Segment reached optimal threshold of 13 pieces. Sep 8 15:07:39.484317854 (49h39m15.523890964s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484413617 (49h39m15.523986727s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484477289 (49h39m15.524050409s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection Sep 8 15:07:39.484496956 (49h39m15.524070076s) [seg_pos={0 0}, upload=43630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:07:39.484556120 (49h39m15.52412924s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:07:39.484560298 (49h39m15.524133418s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:07:39.799359894 (49h39m15.838933014s) [upload=43630]: Done uploading segments... Sep 8 15:07:39.799379402 (49h39m15.838952522s) [upload=43630]: Upload finished. err: <nil> Duration: 10.523055638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [......XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX....] duration: 7.270719023s FAIL (2025-09-08 15:10:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:07:35.933429542 (49h39m11.973002662s) [upload=43630]: Starting upload Sep 8 15:07:35.933522249 (49h39m11.973095369s) [upload=43630]: Done starting upload Sep 8 15:07:35.933619736 (49h39m11.973192856s) [upload=43630]: Uploading segments... Sep 8 15:07:35.933815170 (49h39m11.97338834s) [upload=43630]: Got next segment. Inline: false Sep 8 15:07:35.933860427 (49h39m11.973433537s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:07:35.933873071 (49h39m11.973446191s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:07:36.438715616 (49h39m12.478288736s) [seg_pos={0 0}, upload=43630]: Begin upload segment... Sep 8 15:07:36.438762556 (49h39m12.478335677s) [seg_pos={0 0}, upload=43630]: Done begin upload segment. Sep 8 15:07:36.438787144 (49h39m12.478360264s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.438837350 (49h39m12.47841047s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439289214 (49h39m12.478862334s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439431577 (49h39m12.479004697s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439568549 (49h39m12.479141669s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439761128 (49h39m12.479334248s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439880967 (49h39m12.479454087s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440178376 (49h39m12.479751496s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440239102 (49h39m12.479812222s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440752444 (49h39m12.480325584s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440898734 (49h39m12.480471884s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441275796 (49h39m12.480848916s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441730656 (49h39m12.481303786s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.442853735 (49h39m12.482426865s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443494030 (49h39m12.48306716s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443836766 (49h39m12.483409886s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:38.304825086 (49h39m14.344398206s) [upload=43630]: Next returned nil segment Sep 8 15:07:38.304843241 (49h39m14.344416361s) [upload=43630]: Waiting for error group managing segments... Sep 8 15:07:38.645664021 (49h39m14.685237141s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.650928124 (49h39m14.690501234s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.655750381 (49h39m14.695323501s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.669759469 (49h39m14.709332589s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.672660270 (49h39m14.71223339s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.711041505 (49h39m14.750614625s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.727027635 (49h39m14.766600755s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.792035711 (49h39m14.831608831s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.898063071 (49h39m14.937636192s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.008776007 (49h39m15.048349137s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.100047661 (49h39m15.139620781s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.241123978 (49h39m15.280697128s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484182555 (49h39m15.523755675s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484207583 (49h39m15.523780703s) [seg_pos={0 0}, upload=43630]: Segment reached optimal threshold of 13 pieces. Sep 8 15:07:39.484317854 (49h39m15.523890964s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484413617 (49h39m15.523986727s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484477289 (49h39m15.524050409s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection Sep 8 15:07:39.484496956 (49h39m15.524070076s) [seg_pos={0 0}, upload=43630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:07:39.484556120 (49h39m15.52412924s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:07:39.484560298 (49h39m15.524133418s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:07:39.799359894 (49h39m15.838933014s) [upload=43630]: Done uploading segments... Sep 8 15:07:39.799379402 (49h39m15.838952522s) [upload=43630]: Upload finished. err: <nil> Duration: 7.270771574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.......XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX...] duration: 12.919956558s FAIL (2025-09-08 15:10:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:07:35.933429542 (49h39m11.973002662s) [upload=43630]: Starting upload Sep 8 15:07:35.933522249 (49h39m11.973095369s) [upload=43630]: Done starting upload Sep 8 15:07:35.933619736 (49h39m11.973192856s) [upload=43630]: Uploading segments... Sep 8 15:07:35.933815170 (49h39m11.97338834s) [upload=43630]: Got next segment. Inline: false Sep 8 15:07:35.933860427 (49h39m11.973433537s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:07:35.933873071 (49h39m11.973446191s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:07:36.438715616 (49h39m12.478288736s) [seg_pos={0 0}, upload=43630]: Begin upload segment... Sep 8 15:07:36.438762556 (49h39m12.478335677s) [seg_pos={0 0}, upload=43630]: Done begin upload segment. Sep 8 15:07:36.438787144 (49h39m12.478360264s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.438837350 (49h39m12.47841047s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439289214 (49h39m12.478862334s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439431577 (49h39m12.479004697s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439568549 (49h39m12.479141669s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439761128 (49h39m12.479334248s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439880967 (49h39m12.479454087s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440178376 (49h39m12.479751496s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440239102 (49h39m12.479812222s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440752444 (49h39m12.480325584s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440898734 (49h39m12.480471884s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441275796 (49h39m12.480848916s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441730656 (49h39m12.481303786s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.442853735 (49h39m12.482426865s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443494030 (49h39m12.48306716s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443836766 (49h39m12.483409886s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:38.304825086 (49h39m14.344398206s) [upload=43630]: Next returned nil segment Sep 8 15:07:38.304843241 (49h39m14.344416361s) [upload=43630]: Waiting for error group managing segments... Sep 8 15:07:38.645664021 (49h39m14.685237141s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.650928124 (49h39m14.690501234s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.655750381 (49h39m14.695323501s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.669759469 (49h39m14.709332589s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.672660270 (49h39m14.71223339s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.711041505 (49h39m14.750614625s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.727027635 (49h39m14.766600755s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.792035711 (49h39m14.831608831s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.898063071 (49h39m14.937636192s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.008776007 (49h39m15.048349137s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.100047661 (49h39m15.139620781s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.241123978 (49h39m15.280697128s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484182555 (49h39m15.523755675s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484207583 (49h39m15.523780703s) [seg_pos={0 0}, upload=43630]: Segment reached optimal threshold of 13 pieces. Sep 8 15:07:39.484317854 (49h39m15.523890964s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484413617 (49h39m15.523986727s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484477289 (49h39m15.524050409s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection Sep 8 15:07:39.484496956 (49h39m15.524070076s) [seg_pos={0 0}, upload=43630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:07:39.484556120 (49h39m15.52412924s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:07:39.484560298 (49h39m15.524133418s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:07:39.799359894 (49h39m15.838933014s) [upload=43630]: Done uploading segments... Sep 8 15:07:39.799379402 (49h39m15.838952522s) [upload=43630]: Upload finished. err: <nil> Duration: 12.919985663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [........XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX..] duration: 22.388783566s FAIL (2025-09-08 15:09:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:07:35.933429542 (49h39m11.973002662s) [upload=43630]: Starting upload Sep 8 15:07:35.933522249 (49h39m11.973095369s) [upload=43630]: Done starting upload Sep 8 15:07:35.933619736 (49h39m11.973192856s) [upload=43630]: Uploading segments... Sep 8 15:07:35.933815170 (49h39m11.97338834s) [upload=43630]: Got next segment. Inline: false Sep 8 15:07:35.933860427 (49h39m11.973433537s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:07:35.933873071 (49h39m11.973446191s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:07:36.438715616 (49h39m12.478288736s) [seg_pos={0 0}, upload=43630]: Begin upload segment... Sep 8 15:07:36.438762556 (49h39m12.478335677s) [seg_pos={0 0}, upload=43630]: Done begin upload segment. Sep 8 15:07:36.438787144 (49h39m12.478360264s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.438837350 (49h39m12.47841047s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439289214 (49h39m12.478862334s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439431577 (49h39m12.479004697s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439568549 (49h39m12.479141669s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439761128 (49h39m12.479334248s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439880967 (49h39m12.479454087s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440178376 (49h39m12.479751496s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440239102 (49h39m12.479812222s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440752444 (49h39m12.480325584s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440898734 (49h39m12.480471884s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441275796 (49h39m12.480848916s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441730656 (49h39m12.481303786s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.442853735 (49h39m12.482426865s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443494030 (49h39m12.48306716s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443836766 (49h39m12.483409886s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:38.304825086 (49h39m14.344398206s) [upload=43630]: Next returned nil segment Sep 8 15:07:38.304843241 (49h39m14.344416361s) [upload=43630]: Waiting for error group managing segments... Sep 8 15:07:38.645664021 (49h39m14.685237141s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.650928124 (49h39m14.690501234s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.655750381 (49h39m14.695323501s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.669759469 (49h39m14.709332589s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.672660270 (49h39m14.71223339s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.711041505 (49h39m14.750614625s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.727027635 (49h39m14.766600755s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.792035711 (49h39m14.831608831s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.898063071 (49h39m14.937636192s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.008776007 (49h39m15.048349137s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.100047661 (49h39m15.139620781s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.241123978 (49h39m15.280697128s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484182555 (49h39m15.523755675s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484207583 (49h39m15.523780703s) [seg_pos={0 0}, upload=43630]: Segment reached optimal threshold of 13 pieces. Sep 8 15:07:39.484317854 (49h39m15.523890964s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484413617 (49h39m15.523986727s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484477289 (49h39m15.524050409s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection Sep 8 15:07:39.484496956 (49h39m15.524070076s) [seg_pos={0 0}, upload=43630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:07:39.484556120 (49h39m15.52412924s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:07:39.484560298 (49h39m15.524133418s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:07:39.799359894 (49h39m15.838933014s) [upload=43630]: Done uploading segments... Sep 8 15:07:39.799379402 (49h39m15.838952522s) [upload=43630]: Upload finished. err: <nil> Duration: 22.388812131s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX.] duration: 7.377946222s FAIL (2025-09-08 15:08:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:07:35.933429542 (49h39m11.973002662s) [upload=43630]: Starting upload Sep 8 15:07:35.933522249 (49h39m11.973095369s) [upload=43630]: Done starting upload Sep 8 15:07:35.933619736 (49h39m11.973192856s) [upload=43630]: Uploading segments... Sep 8 15:07:35.933815170 (49h39m11.97338834s) [upload=43630]: Got next segment. Inline: false Sep 8 15:07:35.933860427 (49h39m11.973433537s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:07:35.933873071 (49h39m11.973446191s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:07:36.438715616 (49h39m12.478288736s) [seg_pos={0 0}, upload=43630]: Begin upload segment... Sep 8 15:07:36.438762556 (49h39m12.478335677s) [seg_pos={0 0}, upload=43630]: Done begin upload segment. Sep 8 15:07:36.438787144 (49h39m12.478360264s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.438837350 (49h39m12.47841047s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439289214 (49h39m12.478862334s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439431577 (49h39m12.479004697s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439568549 (49h39m12.479141669s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439761128 (49h39m12.479334248s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439880967 (49h39m12.479454087s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440178376 (49h39m12.479751496s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440239102 (49h39m12.479812222s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440752444 (49h39m12.480325584s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440898734 (49h39m12.480471884s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441275796 (49h39m12.480848916s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441730656 (49h39m12.481303786s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.442853735 (49h39m12.482426865s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443494030 (49h39m12.48306716s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443836766 (49h39m12.483409886s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:38.304825086 (49h39m14.344398206s) [upload=43630]: Next returned nil segment Sep 8 15:07:38.304843241 (49h39m14.344416361s) [upload=43630]: Waiting for error group managing segments... Sep 8 15:07:38.645664021 (49h39m14.685237141s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.650928124 (49h39m14.690501234s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.655750381 (49h39m14.695323501s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.669759469 (49h39m14.709332589s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.672660270 (49h39m14.71223339s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.711041505 (49h39m14.750614625s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.727027635 (49h39m14.766600755s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.792035711 (49h39m14.831608831s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.898063071 (49h39m14.937636192s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.008776007 (49h39m15.048349137s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.100047661 (49h39m15.139620781s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.241123978 (49h39m15.280697128s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484182555 (49h39m15.523755675s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484207583 (49h39m15.523780703s) [seg_pos={0 0}, upload=43630]: Segment reached optimal threshold of 13 pieces. Sep 8 15:07:39.484317854 (49h39m15.523890964s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484413617 (49h39m15.523986727s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484477289 (49h39m15.524050409s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection Sep 8 15:07:39.484496956 (49h39m15.524070076s) [seg_pos={0 0}, upload=43630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:07:39.484556120 (49h39m15.52412924s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:07:39.484560298 (49h39m15.524133418s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:07:39.799359894 (49h39m15.838933014s) [upload=43630]: Done uploading segments... Sep 8 15:07:39.799379402 (49h39m15.838952522s) [upload=43630]: Upload finished. err: <nil> Duration: 7.377970539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...XX] duration: 30.00262438s FAIL (2025-09-08 15:08:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:07:35.933429542 (49h39m11.973002662s) [upload=43630]: Starting upload Sep 8 15:07:35.933522249 (49h39m11.973095369s) [upload=43630]: Done starting upload Sep 8 15:07:35.933619736 (49h39m11.973192856s) [upload=43630]: Uploading segments... Sep 8 15:07:35.933815170 (49h39m11.97338834s) [upload=43630]: Got next segment. Inline: false Sep 8 15:07:35.933860427 (49h39m11.973433537s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:07:35.933873071 (49h39m11.973446191s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:07:36.438715616 (49h39m12.478288736s) [seg_pos={0 0}, upload=43630]: Begin upload segment... Sep 8 15:07:36.438762556 (49h39m12.478335677s) [seg_pos={0 0}, upload=43630]: Done begin upload segment. Sep 8 15:07:36.438787144 (49h39m12.478360264s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.438837350 (49h39m12.47841047s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439289214 (49h39m12.478862334s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439431577 (49h39m12.479004697s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439568549 (49h39m12.479141669s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439761128 (49h39m12.479334248s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.439880967 (49h39m12.479454087s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440178376 (49h39m12.479751496s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440239102 (49h39m12.479812222s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440752444 (49h39m12.480325584s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.440898734 (49h39m12.480471884s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441275796 (49h39m12.480848916s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.441730656 (49h39m12.481303786s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.442853735 (49h39m12.482426865s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443494030 (49h39m12.48306716s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:36.443836766 (49h39m12.483409886s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Uploading piece... Sep 8 15:07:38.304825086 (49h39m14.344398206s) [upload=43630]: Next returned nil segment Sep 8 15:07:38.304843241 (49h39m14.344416361s) [upload=43630]: Waiting for error group managing segments... Sep 8 15:07:38.645664021 (49h39m14.685237141s) [noise=true , address=45.91.116.9:4263 , piece_id=I4IUQH3GCOFZ7PSB4YHJS76ACMY3L3R3C3MTU4T3LGHNSHSIY73A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.650928124 (49h39m14.690501234s) [noise=true , address=77.90.31.16:4734 , piece_id=7AQP5BZLI72FIXJNCNJPQOVMU6KW5OZ5PPLRAJTXKYMRT2EZLJXA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.655750381 (49h39m14.695323501s) [noise=true , address=77.90.31.17:4770 , piece_id=2OHXXAAOENJYG7VKDE4OC3Q5RPS677OKHPPLUJX2EBRI3LVZGW6A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.669759469 (49h39m14.709332589s) [noise=true , address=77.90.31.10:4519 , piece_id=KCZ7LISJ4MXKE2WDNLJTN5IM5VDF4BHSGYUH2HR7BOGOBNZ7DY5A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.672660270 (49h39m14.71223339s) [noise=true , address=45.91.116.11:4118 , piece_id=4YOYFAW4DXUYOAUE6WJK2RCKW5ZGIUK4QRXFZSURYO22SGRGCBJQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.711041505 (49h39m14.750614625s) [noise=true , address=45.91.116.12:4225 , piece_id=YNQZAJC5Q2STMILEAMO5KEIK2ONVGSQAJIXRVM6TUZ5IQMWYKCQQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.727027635 (49h39m14.766600755s) [noise=true , address=77.90.31.18:4807 , piece_id=RTYCBJQZC27DB6T3KT6M7L5MXPPSDS3BY5HWTRCT3LOQNCBBIEAA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.792035711 (49h39m14.831608831s) [noise=true , address=45.91.116.7:4011 , piece_id=4PME4NENKQF5TYS6MBCBPVIVC26TFME4H622I2D63X37ZSQ5SCAQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:38.898063071 (49h39m14.937636192s) [noise=true , address=77.90.31.21:4878 , piece_id=TMHJ6VJEYMBM7TBXYR7TR74CMTANLER3E7LI4FKO2MIREYLINMSQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.008776007 (49h39m15.048349137s) [noise=true , address=145.79.227.15:4873 , piece_id=7L2FZIPB77YAZHJ5UO5WLHBXVFG2W4MVPLJQBDMP6R3RFFMJTFOA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.100047661 (49h39m15.139620781s) [noise=true , address=45.91.116.8:4048 , piece_id=BWI5LTWH3JXLTPHYRU62H2Z7FMHXFJVCPSDRW4RTQ3G5OD5E2HTA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.241123978 (49h39m15.280697128s) [noise=true , address=45.91.116.5:4299 , piece_id=IDUAC7H6M6JCP3W2IZ46TGIKO7S6RM46KM7LDTNNKRWDYLCOOVEA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484182555 (49h39m15.523755675s) [noise=true , address=77.90.31.19:4842 , piece_id=GHW4DKOARCYS3J6XLNDRDD47W447BPNBYPVJ673T5K3ITT3QIJPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: <nil> Sep 8 15:07:39.484207583 (49h39m15.523780703s) [seg_pos={0 0}, upload=43630]: Segment reached optimal threshold of 13 pieces. Sep 8 15:07:39.484317854 (49h39m15.523890964s) [noise=true , address=145.79.227.17:4972 , piece_id=2LPTT2HFI3BM7TFUDHMOC3SITSX7HD3P2OKOMCLS4A7E4WU6AD3A, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484413617 (49h39m15.523986727s) [noise=true , address=45.91.116.14:4190 , piece_id=HXJSTDU2HCWL247FWXWFYYG56KZCR4QCWJLMPD4QQRLNPVGZJAPQ, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:39.484477289 (49h39m15.524050409s) [noise=true , address=45.91.116.13:4153 , piece_id=CICY7E32YTPNGNVGR7AYYMF3KMV6TORAX74H4E53RG67FMKYSHJA, seg_pos={0 0}, upload=43630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57310->45.91.116.13:4153: use of closed network connection Sep 8 15:07:39.484496956 (49h39m15.524070076s) [seg_pos={0 0}, upload=43630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:07:39.484556120 (49h39m15.52412924s) [upload=43630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:07:39.484560298 (49h39m15.524133418s) [upload=43630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:07:39.799359894 (49h39m15.838933014s) [upload=43630]: Done uploading segments... Sep 8 15:07:39.799379402 (49h39m15.838952522s) [upload=43630]: Upload finished. err: <nil> Duration: 30.002654598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...X] duration: 30.001268884s FAIL (2025-09-08 15:07:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:07:01.984232686 (49h38m38.023805806s) [upload=43620]: Starting upload Sep 8 15:07:01.984296207 (49h38m38.023869327s) [upload=43620]: Done starting upload Sep 8 15:07:01.984372794 (49h38m38.023945934s) [upload=43620]: Uploading segments... Sep 8 15:07:01.984462345 (49h38m38.024035465s) [upload=43620]: Got next segment. Inline: false Sep 8 15:07:01.984492032 (49h38m38.024065152s) [upload=43620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:07:01.984497082 (49h38m38.024070202s) [upload=43620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:07:02.490219431 (49h38m38.529792551s) [seg_pos={0 0}, upload=43620]: Begin upload segment... Sep 8 15:07:02.490262484 (49h38m38.529835614s) [seg_pos={0 0}, upload=43620]: Done begin upload segment. Sep 8 15:07:02.490285838 (49h38m38.529858958s) [noise=true , address=45.91.116.9:4263 , piece_id=E4BJF5VDPXOHQ6ZB2WZ4UOKF3PTIQKU3226I42PY7SNCU6N7P56Q, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.490400348 (49h38m38.529973468s) [noise=true , address=45.91.116.5:4298 , piece_id=TDYJMW5O7KQYT72FBZNXQBH5JALVK7Y2FMZA5M4V3PG6QVNNZX5Q, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.490702756 (49h38m38.530275876s) [noise=true , address=145.79.227.15:4875 , piece_id=3GYAZGY3Q6HEKBQEO6QV63K67MTO27X3MZZULDYNL7RQO7MFVYYQ, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.490855037 (49h38m38.530428167s) [noise=true , address=45.91.116.4:4939 , piece_id=BNUTTOL75OT3ZE2UX6FMJLQGXTAS4ITH575I43EST6LJ5XPIIHUA, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.491098132 (49h38m38.530671252s) [noise=true , address=77.90.31.17:4772 , piece_id=C6OAJLN2KRKVMEPABHJGLZNKEFIWVDG3Q35VWE2LBH3EYPBX6CZQ, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.491255404 (49h38m38.530828534s) [noise=true , address=77.90.31.13:4626 , piece_id=WCZRLVGYR74HUG55SR3XFPKEI33TENNOUERP3DWMVVI4NTTAMN4A, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.491344644 (49h38m38.530917774s) [noise=true , address=45.91.116.10:4083 , piece_id=5DTG723S4T2Q6MWBDMY2I6L3DQGXJWMA5DKSURZ5PUUQO2QDFA7A, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.491598811 (49h38m38.531171931s) [noise=true , address=145.79.227.11:4731 , piece_id=YOMQSFCDSXLGMPNIGUGXWTYVX7XLAZDF6XRIK3JWMTRBZYZVIRDA, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.491735683 (49h38m38.531308813s) [noise=true , address=145.79.227.19:4079 , piece_id=UJXQCZS34B5KSU27D3WQWCABCBOUDGG7VZUILUSKDESDIDWW23GA, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.491881802 (49h38m38.531454922s) [noise=true , address=45.91.116.6:4973 , piece_id=5NDRMU32F6BB7MF6FSDMBPMQ3JCZSA73326NN325UUDCLEHZDKVA, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.492038322 (49h38m38.531611442s) [noise=true , address=77.90.31.19:4841 , piece_id=YVTQHZBSGSYK5CPL7RPFMYPABYZXF5C5TB2LGBUMGH44P66O7DUQ, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.492147180 (49h38m38.5317203s) [noise=true , address=45.91.116.7:4011 , piece_id=AWU7TTJ73KKWZMLXHN3YUFHRS3AUXFLBOEJ352A2IPSLHAGM7MEA, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.492285305 (49h38m38.531858425s) [noise=true , address=45.91.116.14:4191 , piece_id=ZI4AJEFA7RK3JFSBPN5USOEDDIEEHGZDFCX2G6BE6F3WT54Y5SCQ, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.492405584 (49h38m38.531978704s) [noise=true , address=77.90.31.18:4805 , piece_id=TNRGZ2DQW5LLYQ2BU5SUGS3ABENEL34CAYHFE6TG6Q3LJOI7HAVA, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.492557766 (49h38m38.532130896s) [noise=true , address=145.79.227.13:4804 , piece_id=G5DORB2HSN7WQD5EQAKSM5GAUJGQULCSPA3ZG6GY2ZEKBNQODJFA, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:02.492805720 (49h38m38.53237884s) [noise=true , address=77.90.31.10:4518 , piece_id=NBB3Y2P24LBPBHITWEZUV5WDOVXHF47GO73I4WP422Q75PJYLSOA, seg_pos={0 0}, upload=43620]: Uploading piece... Sep 8 15:07:04.340415410 (49h38m40.37998853s) [upload=43620]: Next returned nil segment Sep 8 15:07:04.340432032 (49h38m40.380005152s) [upload=43620]: Waiting for error group managing segments... Sep 8 15:07:04.679415637 (49h38m40.718988757s) [noise=true , address=45.91.116.4:4939 , piece_id=BNUTTOL75OT3ZE2UX6FMJLQGXTAS4ITH575I43EST6LJ5XPIIHUA, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.686282617 (49h38m40.725855737s) [noise=true , address=45.91.116.10:4083 , piece_id=5DTG723S4T2Q6MWBDMY2I6L3DQGXJWMA5DKSURZ5PUUQO2QDFA7A, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.691061732 (49h38m40.730634852s) [noise=true , address=45.91.116.7:4011 , piece_id=AWU7TTJ73KKWZMLXHN3YUFHRS3AUXFLBOEJ352A2IPSLHAGM7MEA, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.738009299 (49h38m40.777582419s) [noise=true , address=77.90.31.19:4841 , piece_id=YVTQHZBSGSYK5CPL7RPFMYPABYZXF5C5TB2LGBUMGH44P66O7DUQ, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.751984311 (49h38m40.791557441s) [noise=true , address=45.91.116.14:4191 , piece_id=ZI4AJEFA7RK3JFSBPN5USOEDDIEEHGZDFCX2G6BE6F3WT54Y5SCQ, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.785862189 (49h38m40.825435309s) [noise=true , address=45.91.116.6:4973 , piece_id=5NDRMU32F6BB7MF6FSDMBPMQ3JCZSA73326NN325UUDCLEHZDKVA, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.813187186 (49h38m40.852760306s) [noise=true , address=45.91.116.5:4298 , piece_id=TDYJMW5O7KQYT72FBZNXQBH5JALVK7Y2FMZA5M4V3PG6QVNNZX5Q, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.858206503 (49h38m40.897779623s) [noise=true , address=45.91.116.9:4263 , piece_id=E4BJF5VDPXOHQ6ZB2WZ4UOKF3PTIQKU3226I42PY7SNCU6N7P56Q, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.979835547 (49h38m41.019408667s) [noise=true , address=77.90.31.17:4772 , piece_id=C6OAJLN2KRKVMEPABHJGLZNKEFIWVDG3Q35VWE2LBH3EYPBX6CZQ, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:04.998113925 (49h38m41.037687045s) [noise=true , address=77.90.31.10:4518 , piece_id=NBB3Y2P24LBPBHITWEZUV5WDOVXHF47GO73I4WP422Q75PJYLSOA, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:05.269311707 (49h38m41.308884827s) [noise=true , address=145.79.227.19:4079 , piece_id=UJXQCZS34B5KSU27D3WQWCABCBOUDGG7VZUILUSKDESDIDWW23GA, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:05.305978972 (49h38m41.345552102s) [noise=true , address=145.79.227.11:4731 , piece_id=YOMQSFCDSXLGMPNIGUGXWTYVX7XLAZDF6XRIK3JWMTRBZYZVIRDA, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:05.329442962 (49h38m41.369016082s) [noise=true , address=145.79.227.15:4875 , piece_id=3GYAZGY3Q6HEKBQEO6QV63K67MTO27X3MZZULDYNL7RQO7MFVYYQ, seg_pos={0 0}, upload=43620]: Done uploading piece. err: <nil> Sep 8 15:07:05.329471347 (49h38m41.369044467s) [seg_pos={0 0}, upload=43620]: Segment reached optimal threshold of 13 pieces. Sep 8 15:07:05.329594101 (49h38m41.369167231s) [noise=true , address=77.90.31.18:4805 , piece_id=TNRGZ2DQW5LLYQ2BU5SUGS3ABENEL34CAYHFE6TG6Q3LJOI7HAVA, seg_pos={0 0}, upload=43620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:05.329670227 (49h38m41.369243347s) [noise=true , address=145.79.227.13:4804 , piece_id=G5DORB2HSN7WQD5EQAKSM5GAUJGQULCSPA3ZG6GY2ZEKBNQODJFA, seg_pos={0 0}, upload=43620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:05.329678182 (49h38m41.369251312s) [noise=true , address=77.90.31.13:4626 , piece_id=WCZRLVGYR74HUG55SR3XFPKEI33TENNOUERP3DWMVVI4NTTAMN4A, seg_pos={0 0}, upload=43620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 8 15:07:05.329717728 (49h38m41.369290848s) [seg_pos={0 0}, upload=43620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:07:05.329821656 (49h38m41.369394776s) [upload=43620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:07:05.329826546 (49h38m41.369399666s) [upload=43620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:07:05.649549365 (49h38m41.689122485s) [upload=43620]: Done uploading segments... Sep 8 15:07:05.649589982 (49h38m41.689163102s) [upload=43620]: Upload finished. err: <nil> Duration: 30.001294173s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX...] duration: 22.428025973s FAIL (2025-09-08 15:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:04:09.067168594 (49h35m45.106741714s) [upload=43582]: Starting upload Sep 8 15:04:09.067255820 (49h35m45.106828941s) [upload=43582]: Done starting upload Sep 8 15:04:09.067393424 (49h35m45.106966544s) [upload=43582]: Uploading segments... Sep 8 15:04:09.067722524 (49h35m45.107295655s) [upload=43582]: Got next segment. Inline: false Sep 8 15:04:09.067779533 (49h35m45.107352653s) [upload=43582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:04:09.067787619 (49h35m45.107360739s) [upload=43582]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:04:09.720868196 (49h35m45.760441316s) [seg_pos={0 0}, upload=43582]: Begin upload segment... Sep 8 15:04:09.720914575 (49h35m45.760487685s) [seg_pos={0 0}, upload=43582]: Done begin upload segment. Sep 8 15:04:09.720936647 (49h35m45.760509767s) [noise=true , address=45.91.116.11:4119 , piece_id=FDYHIM3K4OTEZNE26NJEOUYS557M4LAWZZHRJXOFNIIPLCCK3WLA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.720981282 (49h35m45.760554402s) [noise=true , address=77.90.31.12:4589 , piece_id=L4G4E5TA2367Z4JJAKXW2FUDS63EFO45IVTFPTSYRHEHM343IIVQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721338817 (49h35m45.760911937s) [noise=true , address=145.79.227.5:4223 , piece_id=JVLSJUU7ZRVEQRVCQ72YPMGRTK5YJ3JSM7VDNYKJTKD4CVMU5HVQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721453807 (49h35m45.761026917s) [noise=true , address=45.91.116.12:4226 , piece_id=YLVKJKCW4I3CEMNBWXD6EY5VFKHPY6S47XCN2B2U5HBBQCIYTDFQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721709886 (49h35m45.761283007s) [noise=true , address=45.91.116.10:4083 , piece_id=PF5JIMDLTCZW6DUDHWGCKXHINN4XFPQTYGOOTXYY5TX4RY4V33CQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721739383 (49h35m45.761312503s) [noise=true , address=145.79.227.11:4732 , piece_id=IJUFZAP3FJCAXHVYX5R6JXIXAJW5WT6PDMKKISY4UFQTACICQETA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722075306 (49h35m45.761648426s) [noise=true , address=45.91.116.6:4974 , piece_id=V7YJ3EK7HMQ3TL6PUI5G6CIUSJ3S34J2PNM5YVWINZ4KNRNJ5ZNA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722081648 (49h35m45.761654768s) [noise=true , address=77.90.31.15:4698 , piece_id=G4WUJNJ4WTYZ2DPXR43GKF7MV7UO556U4NJHDYC5DG6MAQVZDAXA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722361844 (49h35m45.761934974s) [noise=true , address=77.90.31.13:4626 , piece_id=HHGUHSIZ7AYBHMTCHIGTRCXKGE6Q2BEIQOK6VZP4272AP4PPQRBQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722382684 (49h35m45.761955804s) [noise=true , address=77.90.31.16:4735 , piece_id=2ZWDPHASOOXGGG6UR3M6PNF254GKE7NSX6DVZ4VANSD4L3IHDGLA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722615309 (49h35m45.762188419s) [noise=true , address=45.91.116.13:4154 , piece_id=F4TC2LFMBGFWY27B23FGRIZTBC7C6XHKJJQJRYPGLU7GGIBUDHQA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722676467 (49h35m45.762249587s) [noise=true , address=145.79.227.19:4077 , piece_id=O625OURQUHKHNQV3BDKU5YXZCOB3N52GCBIJ2X2IWKFYWKZLW7MQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722862051 (49h35m45.762435161s) [noise=true , address=45.91.116.9:4261 , piece_id=C6W6JOJIR24TT2UWYLOATPZ4TNFUNEVPCKQWSFXBXRA7SY43RLNA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723039331 (49h35m45.762612451s) [noise=true , address=77.90.31.16:4733 , piece_id=UA4AVB43YXEAPC2B3CGT7UM62TVLLRFUYNDP7MGSC43UM7X3AVZQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723175872 (49h35m45.762748992s) [noise=true , address=45.91.116.8:4045 , piece_id=CKN7KZJ25BYNHLZZLPK3CVTDVHPCTTANZI5VMUBUD6JCDLRTF7FQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723401984 (49h35m45.762975104s) [noise=true , address=45.91.116.13:4155 , piece_id=E4D6MGGU34LEXME5ZX3EOVRJUULA5HMA7S2EYWSL54JU5UUCL23Q, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:11.611629719 (49h35m47.651202839s) [upload=43582]: Next returned nil segment Sep 8 15:04:11.611667482 (49h35m47.651240602s) [upload=43582]: Waiting for error group managing segments... Sep 8 15:04:12.093530495 (49h35m48.133103615s) [noise=true , address=145.79.227.11:4732 , piece_id=IJUFZAP3FJCAXHVYX5R6JXIXAJW5WT6PDMKKISY4UFQTACICQETA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.107467363 (49h35m48.147040483s) [noise=true , address=45.91.116.10:4083 , piece_id=PF5JIMDLTCZW6DUDHWGCKXHINN4XFPQTYGOOTXYY5TX4RY4V33CQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.115415572 (49h35m48.154988693s) [noise=true , address=45.91.116.12:4226 , piece_id=YLVKJKCW4I3CEMNBWXD6EY5VFKHPY6S47XCN2B2U5HBBQCIYTDFQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.115537536 (49h35m48.155110656s) [noise=true , address=45.91.116.9:4261 , piece_id=C6W6JOJIR24TT2UWYLOATPZ4TNFUNEVPCKQWSFXBXRA7SY43RLNA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.117043849 (49h35m48.156616969s) [noise=true , address=77.90.31.16:4735 , piece_id=2ZWDPHASOOXGGG6UR3M6PNF254GKE7NSX6DVZ4VANSD4L3IHDGLA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.138127795 (49h35m48.177700915s) [noise=true , address=45.91.116.8:4045 , piece_id=CKN7KZJ25BYNHLZZLPK3CVTDVHPCTTANZI5VMUBUD6JCDLRTF7FQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.184499549 (49h35m48.224072669s) [noise=true , address=77.90.31.16:4733 , piece_id=UA4AVB43YXEAPC2B3CGT7UM62TVLLRFUYNDP7MGSC43UM7X3AVZQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.213652265 (49h35m48.253225375s) [noise=true , address=145.79.227.5:4223 , piece_id=JVLSJUU7ZRVEQRVCQ72YPMGRTK5YJ3JSM7VDNYKJTKD4CVMU5HVQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.215540678 (49h35m48.255113798s) [noise=true , address=145.79.227.19:4077 , piece_id=O625OURQUHKHNQV3BDKU5YXZCOB3N52GCBIJ2X2IWKFYWKZLW7MQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.228685261 (49h35m48.268258381s) [noise=true , address=77.90.31.13:4626 , piece_id=HHGUHSIZ7AYBHMTCHIGTRCXKGE6Q2BEIQOK6VZP4272AP4PPQRBQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.438864270 (49h35m48.47843739s) [noise=true , address=45.91.116.6:4974 , piece_id=V7YJ3EK7HMQ3TL6PUI5G6CIUSJ3S34J2PNM5YVWINZ4KNRNJ5ZNA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.515721171 (49h35m48.555294291s) [noise=true , address=77.90.31.15:4698 , piece_id=G4WUJNJ4WTYZ2DPXR43GKF7MV7UO556U4NJHDYC5DG6MAQVZDAXA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.599106514 (49h35m48.638679634s) [noise=true , address=77.90.31.12:4589 , piece_id=L4G4E5TA2367Z4JJAKXW2FUDS63EFO45IVTFPTSYRHEHM343IIVQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.599125811 (49h35m48.638698931s) [seg_pos={0 0}, upload=43582]: Segment reached optimal threshold of 13 pieces. Sep 8 15:04:12.599349329 (49h35m48.638922439s) [noise=true , address=45.91.116.11:4119 , piece_id=FDYHIM3K4OTEZNE26NJEOUYS557M4LAWZZHRJXOFNIIPLCCK3WLA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection Sep 8 15:04:12.599402460 (49h35m48.63897558s) [noise=true , address=45.91.116.13:4155 , piece_id=E4D6MGGU34LEXME5ZX3EOVRJUULA5HMA7S2EYWSL54JU5UUCL23Q, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection Sep 8 15:04:12.599469418 (49h35m48.639042538s) [noise=true , address=45.91.116.13:4154 , piece_id=F4TC2LFMBGFWY27B23FGRIZTBC7C6XHKJJQJRYPGLU7GGIBUDHQA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection Sep 8 15:04:12.599522730 (49h35m48.63909584s) [seg_pos={0 0}, upload=43582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:04:12.599619876 (49h35m48.639192997s) [upload=43582]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:04:12.599624545 (49h35m48.639197665s) [upload=43582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:04:12.974002441 (49h35m49.013575561s) [upload=43582]: Done uploading segments... Sep 8 15:04:12.974028570 (49h35m49.01360169s) [upload=43582]: Upload finished. err: <nil> Duration: 22.428047716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX..] duration: 19.839049318s FAIL (2025-09-08 15:06:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:04:09.067168594 (49h35m45.106741714s) [upload=43582]: Starting upload Sep 8 15:04:09.067255820 (49h35m45.106828941s) [upload=43582]: Done starting upload Sep 8 15:04:09.067393424 (49h35m45.106966544s) [upload=43582]: Uploading segments... Sep 8 15:04:09.067722524 (49h35m45.107295655s) [upload=43582]: Got next segment. Inline: false Sep 8 15:04:09.067779533 (49h35m45.107352653s) [upload=43582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:04:09.067787619 (49h35m45.107360739s) [upload=43582]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:04:09.720868196 (49h35m45.760441316s) [seg_pos={0 0}, upload=43582]: Begin upload segment... Sep 8 15:04:09.720914575 (49h35m45.760487685s) [seg_pos={0 0}, upload=43582]: Done begin upload segment. Sep 8 15:04:09.720936647 (49h35m45.760509767s) [noise=true , address=45.91.116.11:4119 , piece_id=FDYHIM3K4OTEZNE26NJEOUYS557M4LAWZZHRJXOFNIIPLCCK3WLA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.720981282 (49h35m45.760554402s) [noise=true , address=77.90.31.12:4589 , piece_id=L4G4E5TA2367Z4JJAKXW2FUDS63EFO45IVTFPTSYRHEHM343IIVQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721338817 (49h35m45.760911937s) [noise=true , address=145.79.227.5:4223 , piece_id=JVLSJUU7ZRVEQRVCQ72YPMGRTK5YJ3JSM7VDNYKJTKD4CVMU5HVQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721453807 (49h35m45.761026917s) [noise=true , address=45.91.116.12:4226 , piece_id=YLVKJKCW4I3CEMNBWXD6EY5VFKHPY6S47XCN2B2U5HBBQCIYTDFQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721709886 (49h35m45.761283007s) [noise=true , address=45.91.116.10:4083 , piece_id=PF5JIMDLTCZW6DUDHWGCKXHINN4XFPQTYGOOTXYY5TX4RY4V33CQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721739383 (49h35m45.761312503s) [noise=true , address=145.79.227.11:4732 , piece_id=IJUFZAP3FJCAXHVYX5R6JXIXAJW5WT6PDMKKISY4UFQTACICQETA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722075306 (49h35m45.761648426s) [noise=true , address=45.91.116.6:4974 , piece_id=V7YJ3EK7HMQ3TL6PUI5G6CIUSJ3S34J2PNM5YVWINZ4KNRNJ5ZNA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722081648 (49h35m45.761654768s) [noise=true , address=77.90.31.15:4698 , piece_id=G4WUJNJ4WTYZ2DPXR43GKF7MV7UO556U4NJHDYC5DG6MAQVZDAXA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722361844 (49h35m45.761934974s) [noise=true , address=77.90.31.13:4626 , piece_id=HHGUHSIZ7AYBHMTCHIGTRCXKGE6Q2BEIQOK6VZP4272AP4PPQRBQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722382684 (49h35m45.761955804s) [noise=true , address=77.90.31.16:4735 , piece_id=2ZWDPHASOOXGGG6UR3M6PNF254GKE7NSX6DVZ4VANSD4L3IHDGLA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722615309 (49h35m45.762188419s) [noise=true , address=45.91.116.13:4154 , piece_id=F4TC2LFMBGFWY27B23FGRIZTBC7C6XHKJJQJRYPGLU7GGIBUDHQA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722676467 (49h35m45.762249587s) [noise=true , address=145.79.227.19:4077 , piece_id=O625OURQUHKHNQV3BDKU5YXZCOB3N52GCBIJ2X2IWKFYWKZLW7MQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722862051 (49h35m45.762435161s) [noise=true , address=45.91.116.9:4261 , piece_id=C6W6JOJIR24TT2UWYLOATPZ4TNFUNEVPCKQWSFXBXRA7SY43RLNA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723039331 (49h35m45.762612451s) [noise=true , address=77.90.31.16:4733 , piece_id=UA4AVB43YXEAPC2B3CGT7UM62TVLLRFUYNDP7MGSC43UM7X3AVZQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723175872 (49h35m45.762748992s) [noise=true , address=45.91.116.8:4045 , piece_id=CKN7KZJ25BYNHLZZLPK3CVTDVHPCTTANZI5VMUBUD6JCDLRTF7FQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723401984 (49h35m45.762975104s) [noise=true , address=45.91.116.13:4155 , piece_id=E4D6MGGU34LEXME5ZX3EOVRJUULA5HMA7S2EYWSL54JU5UUCL23Q, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:11.611629719 (49h35m47.651202839s) [upload=43582]: Next returned nil segment Sep 8 15:04:11.611667482 (49h35m47.651240602s) [upload=43582]: Waiting for error group managing segments... Sep 8 15:04:12.093530495 (49h35m48.133103615s) [noise=true , address=145.79.227.11:4732 , piece_id=IJUFZAP3FJCAXHVYX5R6JXIXAJW5WT6PDMKKISY4UFQTACICQETA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.107467363 (49h35m48.147040483s) [noise=true , address=45.91.116.10:4083 , piece_id=PF5JIMDLTCZW6DUDHWGCKXHINN4XFPQTYGOOTXYY5TX4RY4V33CQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.115415572 (49h35m48.154988693s) [noise=true , address=45.91.116.12:4226 , piece_id=YLVKJKCW4I3CEMNBWXD6EY5VFKHPY6S47XCN2B2U5HBBQCIYTDFQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.115537536 (49h35m48.155110656s) [noise=true , address=45.91.116.9:4261 , piece_id=C6W6JOJIR24TT2UWYLOATPZ4TNFUNEVPCKQWSFXBXRA7SY43RLNA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.117043849 (49h35m48.156616969s) [noise=true , address=77.90.31.16:4735 , piece_id=2ZWDPHASOOXGGG6UR3M6PNF254GKE7NSX6DVZ4VANSD4L3IHDGLA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.138127795 (49h35m48.177700915s) [noise=true , address=45.91.116.8:4045 , piece_id=CKN7KZJ25BYNHLZZLPK3CVTDVHPCTTANZI5VMUBUD6JCDLRTF7FQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.184499549 (49h35m48.224072669s) [noise=true , address=77.90.31.16:4733 , piece_id=UA4AVB43YXEAPC2B3CGT7UM62TVLLRFUYNDP7MGSC43UM7X3AVZQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.213652265 (49h35m48.253225375s) [noise=true , address=145.79.227.5:4223 , piece_id=JVLSJUU7ZRVEQRVCQ72YPMGRTK5YJ3JSM7VDNYKJTKD4CVMU5HVQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.215540678 (49h35m48.255113798s) [noise=true , address=145.79.227.19:4077 , piece_id=O625OURQUHKHNQV3BDKU5YXZCOB3N52GCBIJ2X2IWKFYWKZLW7MQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.228685261 (49h35m48.268258381s) [noise=true , address=77.90.31.13:4626 , piece_id=HHGUHSIZ7AYBHMTCHIGTRCXKGE6Q2BEIQOK6VZP4272AP4PPQRBQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.438864270 (49h35m48.47843739s) [noise=true , address=45.91.116.6:4974 , piece_id=V7YJ3EK7HMQ3TL6PUI5G6CIUSJ3S34J2PNM5YVWINZ4KNRNJ5ZNA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.515721171 (49h35m48.555294291s) [noise=true , address=77.90.31.15:4698 , piece_id=G4WUJNJ4WTYZ2DPXR43GKF7MV7UO556U4NJHDYC5DG6MAQVZDAXA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.599106514 (49h35m48.638679634s) [noise=true , address=77.90.31.12:4589 , piece_id=L4G4E5TA2367Z4JJAKXW2FUDS63EFO45IVTFPTSYRHEHM343IIVQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.599125811 (49h35m48.638698931s) [seg_pos={0 0}, upload=43582]: Segment reached optimal threshold of 13 pieces. Sep 8 15:04:12.599349329 (49h35m48.638922439s) [noise=true , address=45.91.116.11:4119 , piece_id=FDYHIM3K4OTEZNE26NJEOUYS557M4LAWZZHRJXOFNIIPLCCK3WLA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection Sep 8 15:04:12.599402460 (49h35m48.63897558s) [noise=true , address=45.91.116.13:4155 , piece_id=E4D6MGGU34LEXME5ZX3EOVRJUULA5HMA7S2EYWSL54JU5UUCL23Q, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection Sep 8 15:04:12.599469418 (49h35m48.639042538s) [noise=true , address=45.91.116.13:4154 , piece_id=F4TC2LFMBGFWY27B23FGRIZTBC7C6XHKJJQJRYPGLU7GGIBUDHQA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection Sep 8 15:04:12.599522730 (49h35m48.63909584s) [seg_pos={0 0}, upload=43582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:04:12.599619876 (49h35m48.639192997s) [upload=43582]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:04:12.599624545 (49h35m48.639197665s) [upload=43582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:04:12.974002441 (49h35m49.013575561s) [upload=43582]: Done uploading segments... Sep 8 15:04:12.974028570 (49h35m49.01360169s) [upload=43582]: Upload finished. err: <nil> Duration: 19.839078832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX.] duration: 23.257697441s FAIL (2025-09-08 15:05:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:04:09.067168594 (49h35m45.106741714s) [upload=43582]: Starting upload Sep 8 15:04:09.067255820 (49h35m45.106828941s) [upload=43582]: Done starting upload Sep 8 15:04:09.067393424 (49h35m45.106966544s) [upload=43582]: Uploading segments... Sep 8 15:04:09.067722524 (49h35m45.107295655s) [upload=43582]: Got next segment. Inline: false Sep 8 15:04:09.067779533 (49h35m45.107352653s) [upload=43582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:04:09.067787619 (49h35m45.107360739s) [upload=43582]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:04:09.720868196 (49h35m45.760441316s) [seg_pos={0 0}, upload=43582]: Begin upload segment... Sep 8 15:04:09.720914575 (49h35m45.760487685s) [seg_pos={0 0}, upload=43582]: Done begin upload segment. Sep 8 15:04:09.720936647 (49h35m45.760509767s) [noise=true , address=45.91.116.11:4119 , piece_id=FDYHIM3K4OTEZNE26NJEOUYS557M4LAWZZHRJXOFNIIPLCCK3WLA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.720981282 (49h35m45.760554402s) [noise=true , address=77.90.31.12:4589 , piece_id=L4G4E5TA2367Z4JJAKXW2FUDS63EFO45IVTFPTSYRHEHM343IIVQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721338817 (49h35m45.760911937s) [noise=true , address=145.79.227.5:4223 , piece_id=JVLSJUU7ZRVEQRVCQ72YPMGRTK5YJ3JSM7VDNYKJTKD4CVMU5HVQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721453807 (49h35m45.761026917s) [noise=true , address=45.91.116.12:4226 , piece_id=YLVKJKCW4I3CEMNBWXD6EY5VFKHPY6S47XCN2B2U5HBBQCIYTDFQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721709886 (49h35m45.761283007s) [noise=true , address=45.91.116.10:4083 , piece_id=PF5JIMDLTCZW6DUDHWGCKXHINN4XFPQTYGOOTXYY5TX4RY4V33CQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721739383 (49h35m45.761312503s) [noise=true , address=145.79.227.11:4732 , piece_id=IJUFZAP3FJCAXHVYX5R6JXIXAJW5WT6PDMKKISY4UFQTACICQETA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722075306 (49h35m45.761648426s) [noise=true , address=45.91.116.6:4974 , piece_id=V7YJ3EK7HMQ3TL6PUI5G6CIUSJ3S34J2PNM5YVWINZ4KNRNJ5ZNA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722081648 (49h35m45.761654768s) [noise=true , address=77.90.31.15:4698 , piece_id=G4WUJNJ4WTYZ2DPXR43GKF7MV7UO556U4NJHDYC5DG6MAQVZDAXA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722361844 (49h35m45.761934974s) [noise=true , address=77.90.31.13:4626 , piece_id=HHGUHSIZ7AYBHMTCHIGTRCXKGE6Q2BEIQOK6VZP4272AP4PPQRBQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722382684 (49h35m45.761955804s) [noise=true , address=77.90.31.16:4735 , piece_id=2ZWDPHASOOXGGG6UR3M6PNF254GKE7NSX6DVZ4VANSD4L3IHDGLA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722615309 (49h35m45.762188419s) [noise=true , address=45.91.116.13:4154 , piece_id=F4TC2LFMBGFWY27B23FGRIZTBC7C6XHKJJQJRYPGLU7GGIBUDHQA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722676467 (49h35m45.762249587s) [noise=true , address=145.79.227.19:4077 , piece_id=O625OURQUHKHNQV3BDKU5YXZCOB3N52GCBIJ2X2IWKFYWKZLW7MQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722862051 (49h35m45.762435161s) [noise=true , address=45.91.116.9:4261 , piece_id=C6W6JOJIR24TT2UWYLOATPZ4TNFUNEVPCKQWSFXBXRA7SY43RLNA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723039331 (49h35m45.762612451s) [noise=true , address=77.90.31.16:4733 , piece_id=UA4AVB43YXEAPC2B3CGT7UM62TVLLRFUYNDP7MGSC43UM7X3AVZQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723175872 (49h35m45.762748992s) [noise=true , address=45.91.116.8:4045 , piece_id=CKN7KZJ25BYNHLZZLPK3CVTDVHPCTTANZI5VMUBUD6JCDLRTF7FQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723401984 (49h35m45.762975104s) [noise=true , address=45.91.116.13:4155 , piece_id=E4D6MGGU34LEXME5ZX3EOVRJUULA5HMA7S2EYWSL54JU5UUCL23Q, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:11.611629719 (49h35m47.651202839s) [upload=43582]: Next returned nil segment Sep 8 15:04:11.611667482 (49h35m47.651240602s) [upload=43582]: Waiting for error group managing segments... Sep 8 15:04:12.093530495 (49h35m48.133103615s) [noise=true , address=145.79.227.11:4732 , piece_id=IJUFZAP3FJCAXHVYX5R6JXIXAJW5WT6PDMKKISY4UFQTACICQETA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.107467363 (49h35m48.147040483s) [noise=true , address=45.91.116.10:4083 , piece_id=PF5JIMDLTCZW6DUDHWGCKXHINN4XFPQTYGOOTXYY5TX4RY4V33CQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.115415572 (49h35m48.154988693s) [noise=true , address=45.91.116.12:4226 , piece_id=YLVKJKCW4I3CEMNBWXD6EY5VFKHPY6S47XCN2B2U5HBBQCIYTDFQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.115537536 (49h35m48.155110656s) [noise=true , address=45.91.116.9:4261 , piece_id=C6W6JOJIR24TT2UWYLOATPZ4TNFUNEVPCKQWSFXBXRA7SY43RLNA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.117043849 (49h35m48.156616969s) [noise=true , address=77.90.31.16:4735 , piece_id=2ZWDPHASOOXGGG6UR3M6PNF254GKE7NSX6DVZ4VANSD4L3IHDGLA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.138127795 (49h35m48.177700915s) [noise=true , address=45.91.116.8:4045 , piece_id=CKN7KZJ25BYNHLZZLPK3CVTDVHPCTTANZI5VMUBUD6JCDLRTF7FQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.184499549 (49h35m48.224072669s) [noise=true , address=77.90.31.16:4733 , piece_id=UA4AVB43YXEAPC2B3CGT7UM62TVLLRFUYNDP7MGSC43UM7X3AVZQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.213652265 (49h35m48.253225375s) [noise=true , address=145.79.227.5:4223 , piece_id=JVLSJUU7ZRVEQRVCQ72YPMGRTK5YJ3JSM7VDNYKJTKD4CVMU5HVQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.215540678 (49h35m48.255113798s) [noise=true , address=145.79.227.19:4077 , piece_id=O625OURQUHKHNQV3BDKU5YXZCOB3N52GCBIJ2X2IWKFYWKZLW7MQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.228685261 (49h35m48.268258381s) [noise=true , address=77.90.31.13:4626 , piece_id=HHGUHSIZ7AYBHMTCHIGTRCXKGE6Q2BEIQOK6VZP4272AP4PPQRBQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.438864270 (49h35m48.47843739s) [noise=true , address=45.91.116.6:4974 , piece_id=V7YJ3EK7HMQ3TL6PUI5G6CIUSJ3S34J2PNM5YVWINZ4KNRNJ5ZNA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.515721171 (49h35m48.555294291s) [noise=true , address=77.90.31.15:4698 , piece_id=G4WUJNJ4WTYZ2DPXR43GKF7MV7UO556U4NJHDYC5DG6MAQVZDAXA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.599106514 (49h35m48.638679634s) [noise=true , address=77.90.31.12:4589 , piece_id=L4G4E5TA2367Z4JJAKXW2FUDS63EFO45IVTFPTSYRHEHM343IIVQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.599125811 (49h35m48.638698931s) [seg_pos={0 0}, upload=43582]: Segment reached optimal threshold of 13 pieces. Sep 8 15:04:12.599349329 (49h35m48.638922439s) [noise=true , address=45.91.116.11:4119 , piece_id=FDYHIM3K4OTEZNE26NJEOUYS557M4LAWZZHRJXOFNIIPLCCK3WLA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection Sep 8 15:04:12.599402460 (49h35m48.63897558s) [noise=true , address=45.91.116.13:4155 , piece_id=E4D6MGGU34LEXME5ZX3EOVRJUULA5HMA7S2EYWSL54JU5UUCL23Q, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection Sep 8 15:04:12.599469418 (49h35m48.639042538s) [noise=true , address=45.91.116.13:4154 , piece_id=F4TC2LFMBGFWY27B23FGRIZTBC7C6XHKJJQJRYPGLU7GGIBUDHQA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection Sep 8 15:04:12.599522730 (49h35m48.63909584s) [seg_pos={0 0}, upload=43582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:04:12.599619876 (49h35m48.639192997s) [upload=43582]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:04:12.599624545 (49h35m48.639197665s) [upload=43582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:04:12.974002441 (49h35m49.013575561s) [upload=43582]: Done uploading segments... Sep 8 15:04:12.974028570 (49h35m49.01360169s) [upload=43582]: Upload finished. err: <nil> Duration: 23.257727589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...XX] duration: 30.001456638s FAIL (2025-09-08 15:04:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:04:09.067168594 (49h35m45.106741714s) [upload=43582]: Starting upload Sep 8 15:04:09.067255820 (49h35m45.106828941s) [upload=43582]: Done starting upload Sep 8 15:04:09.067393424 (49h35m45.106966544s) [upload=43582]: Uploading segments... Sep 8 15:04:09.067722524 (49h35m45.107295655s) [upload=43582]: Got next segment. Inline: false Sep 8 15:04:09.067779533 (49h35m45.107352653s) [upload=43582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:04:09.067787619 (49h35m45.107360739s) [upload=43582]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:04:09.720868196 (49h35m45.760441316s) [seg_pos={0 0}, upload=43582]: Begin upload segment... Sep 8 15:04:09.720914575 (49h35m45.760487685s) [seg_pos={0 0}, upload=43582]: Done begin upload segment. Sep 8 15:04:09.720936647 (49h35m45.760509767s) [noise=true , address=45.91.116.11:4119 , piece_id=FDYHIM3K4OTEZNE26NJEOUYS557M4LAWZZHRJXOFNIIPLCCK3WLA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.720981282 (49h35m45.760554402s) [noise=true , address=77.90.31.12:4589 , piece_id=L4G4E5TA2367Z4JJAKXW2FUDS63EFO45IVTFPTSYRHEHM343IIVQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721338817 (49h35m45.760911937s) [noise=true , address=145.79.227.5:4223 , piece_id=JVLSJUU7ZRVEQRVCQ72YPMGRTK5YJ3JSM7VDNYKJTKD4CVMU5HVQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721453807 (49h35m45.761026917s) [noise=true , address=45.91.116.12:4226 , piece_id=YLVKJKCW4I3CEMNBWXD6EY5VFKHPY6S47XCN2B2U5HBBQCIYTDFQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721709886 (49h35m45.761283007s) [noise=true , address=45.91.116.10:4083 , piece_id=PF5JIMDLTCZW6DUDHWGCKXHINN4XFPQTYGOOTXYY5TX4RY4V33CQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.721739383 (49h35m45.761312503s) [noise=true , address=145.79.227.11:4732 , piece_id=IJUFZAP3FJCAXHVYX5R6JXIXAJW5WT6PDMKKISY4UFQTACICQETA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722075306 (49h35m45.761648426s) [noise=true , address=45.91.116.6:4974 , piece_id=V7YJ3EK7HMQ3TL6PUI5G6CIUSJ3S34J2PNM5YVWINZ4KNRNJ5ZNA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722081648 (49h35m45.761654768s) [noise=true , address=77.90.31.15:4698 , piece_id=G4WUJNJ4WTYZ2DPXR43GKF7MV7UO556U4NJHDYC5DG6MAQVZDAXA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722361844 (49h35m45.761934974s) [noise=true , address=77.90.31.13:4626 , piece_id=HHGUHSIZ7AYBHMTCHIGTRCXKGE6Q2BEIQOK6VZP4272AP4PPQRBQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722382684 (49h35m45.761955804s) [noise=true , address=77.90.31.16:4735 , piece_id=2ZWDPHASOOXGGG6UR3M6PNF254GKE7NSX6DVZ4VANSD4L3IHDGLA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722615309 (49h35m45.762188419s) [noise=true , address=45.91.116.13:4154 , piece_id=F4TC2LFMBGFWY27B23FGRIZTBC7C6XHKJJQJRYPGLU7GGIBUDHQA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722676467 (49h35m45.762249587s) [noise=true , address=145.79.227.19:4077 , piece_id=O625OURQUHKHNQV3BDKU5YXZCOB3N52GCBIJ2X2IWKFYWKZLW7MQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.722862051 (49h35m45.762435161s) [noise=true , address=45.91.116.9:4261 , piece_id=C6W6JOJIR24TT2UWYLOATPZ4TNFUNEVPCKQWSFXBXRA7SY43RLNA, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723039331 (49h35m45.762612451s) [noise=true , address=77.90.31.16:4733 , piece_id=UA4AVB43YXEAPC2B3CGT7UM62TVLLRFUYNDP7MGSC43UM7X3AVZQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723175872 (49h35m45.762748992s) [noise=true , address=45.91.116.8:4045 , piece_id=CKN7KZJ25BYNHLZZLPK3CVTDVHPCTTANZI5VMUBUD6JCDLRTF7FQ, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:09.723401984 (49h35m45.762975104s) [noise=true , address=45.91.116.13:4155 , piece_id=E4D6MGGU34LEXME5ZX3EOVRJUULA5HMA7S2EYWSL54JU5UUCL23Q, seg_pos={0 0}, upload=43582]: Uploading piece... Sep 8 15:04:11.611629719 (49h35m47.651202839s) [upload=43582]: Next returned nil segment Sep 8 15:04:11.611667482 (49h35m47.651240602s) [upload=43582]: Waiting for error group managing segments... Sep 8 15:04:12.093530495 (49h35m48.133103615s) [noise=true , address=145.79.227.11:4732 , piece_id=IJUFZAP3FJCAXHVYX5R6JXIXAJW5WT6PDMKKISY4UFQTACICQETA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.107467363 (49h35m48.147040483s) [noise=true , address=45.91.116.10:4083 , piece_id=PF5JIMDLTCZW6DUDHWGCKXHINN4XFPQTYGOOTXYY5TX4RY4V33CQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.115415572 (49h35m48.154988693s) [noise=true , address=45.91.116.12:4226 , piece_id=YLVKJKCW4I3CEMNBWXD6EY5VFKHPY6S47XCN2B2U5HBBQCIYTDFQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.115537536 (49h35m48.155110656s) [noise=true , address=45.91.116.9:4261 , piece_id=C6W6JOJIR24TT2UWYLOATPZ4TNFUNEVPCKQWSFXBXRA7SY43RLNA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.117043849 (49h35m48.156616969s) [noise=true , address=77.90.31.16:4735 , piece_id=2ZWDPHASOOXGGG6UR3M6PNF254GKE7NSX6DVZ4VANSD4L3IHDGLA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.138127795 (49h35m48.177700915s) [noise=true , address=45.91.116.8:4045 , piece_id=CKN7KZJ25BYNHLZZLPK3CVTDVHPCTTANZI5VMUBUD6JCDLRTF7FQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.184499549 (49h35m48.224072669s) [noise=true , address=77.90.31.16:4733 , piece_id=UA4AVB43YXEAPC2B3CGT7UM62TVLLRFUYNDP7MGSC43UM7X3AVZQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.213652265 (49h35m48.253225375s) [noise=true , address=145.79.227.5:4223 , piece_id=JVLSJUU7ZRVEQRVCQ72YPMGRTK5YJ3JSM7VDNYKJTKD4CVMU5HVQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.215540678 (49h35m48.255113798s) [noise=true , address=145.79.227.19:4077 , piece_id=O625OURQUHKHNQV3BDKU5YXZCOB3N52GCBIJ2X2IWKFYWKZLW7MQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.228685261 (49h35m48.268258381s) [noise=true , address=77.90.31.13:4626 , piece_id=HHGUHSIZ7AYBHMTCHIGTRCXKGE6Q2BEIQOK6VZP4272AP4PPQRBQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.438864270 (49h35m48.47843739s) [noise=true , address=45.91.116.6:4974 , piece_id=V7YJ3EK7HMQ3TL6PUI5G6CIUSJ3S34J2PNM5YVWINZ4KNRNJ5ZNA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.515721171 (49h35m48.555294291s) [noise=true , address=77.90.31.15:4698 , piece_id=G4WUJNJ4WTYZ2DPXR43GKF7MV7UO556U4NJHDYC5DG6MAQVZDAXA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.599106514 (49h35m48.638679634s) [noise=true , address=77.90.31.12:4589 , piece_id=L4G4E5TA2367Z4JJAKXW2FUDS63EFO45IVTFPTSYRHEHM343IIVQ, seg_pos={0 0}, upload=43582]: Done uploading piece. err: <nil> Sep 8 15:04:12.599125811 (49h35m48.638698931s) [seg_pos={0 0}, upload=43582]: Segment reached optimal threshold of 13 pieces. Sep 8 15:04:12.599349329 (49h35m48.638922439s) [noise=true , address=45.91.116.11:4119 , piece_id=FDYHIM3K4OTEZNE26NJEOUYS557M4LAWZZHRJXOFNIIPLCCK3WLA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48394->45.91.116.11:4119: use of closed network connection Sep 8 15:04:12.599402460 (49h35m48.63897558s) [noise=true , address=45.91.116.13:4155 , piece_id=E4D6MGGU34LEXME5ZX3EOVRJUULA5HMA7S2EYWSL54JU5UUCL23Q, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50874->45.91.116.13:4155: use of closed network connection Sep 8 15:04:12.599469418 (49h35m48.639042538s) [noise=true , address=45.91.116.13:4154 , piece_id=F4TC2LFMBGFWY27B23FGRIZTBC7C6XHKJJQJRYPGLU7GGIBUDHQA, seg_pos={0 0}, upload=43582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->45.91.116.13:4154: use of closed network connection Sep 8 15:04:12.599522730 (49h35m48.63909584s) [seg_pos={0 0}, upload=43582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:04:12.599619876 (49h35m48.639192997s) [upload=43582]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:04:12.599624545 (49h35m48.639197665s) [upload=43582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:04:12.974002441 (49h35m49.013575561s) [upload=43582]: Done uploading segments... Sep 8 15:04:12.974028570 (49h35m49.01360169s) [upload=43582]: Upload finished. err: <nil> Duration: 30.00147327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...X] duration: 30.002372828s FAIL (2025-09-08 15:04:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:03:37.116363928 (49h35m13.155937048s) [upload=43572]: Starting upload Sep 8 15:03:37.116427259 (49h35m13.156000379s) [upload=43572]: Done starting upload Sep 8 15:03:37.116519245 (49h35m13.156092365s) [upload=43572]: Uploading segments... Sep 8 15:03:37.116727173 (49h35m13.156300293s) [upload=43572]: Got next segment. Inline: false Sep 8 15:03:37.116756800 (49h35m13.15632992s) [upload=43572]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:03:37.116761959 (49h35m13.156335079s) [upload=43572]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:03:37.390822831 (49h35m13.430395951s) [seg_pos={0 0}, upload=43572]: Begin upload segment... Sep 8 15:03:37.390867677 (49h35m13.430440797s) [seg_pos={0 0}, upload=43572]: Done begin upload segment. Sep 8 15:03:37.390941058 (49h35m13.430514178s) [noise=true , address=45.91.116.9:4263 , piece_id=ODM6ZAICRWMXQGW6NCJOFXAX7MZIJTRIRB3LP6QBONZAA74UMG4A, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.391315403 (49h35m13.430888523s) [noise=true , address=77.90.31.19:4841 , piece_id=5AQEHZ5IP2WLCDHXNIY5JQLDLGAEZTRVWAFXQE3GIULYUDMAHYBA, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.391578978 (49h35m13.431152098s) [noise=true , address=45.91.116.10:4083 , piece_id=EPCSBLYDH3D3FNI3Z2R7RO3HZHJ56MYKMUU2EWF3XRCAQXECAZZQ, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.391834967 (49h35m13.431408087s) [noise=true , address=145.79.227.11:4730 , piece_id=SWJP3EFCY57ZMESBBTMFO27DDZEP6WEKVITLHC4LRKR6KHHSJB2A, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.392146925 (49h35m13.431720035s) [noise=true , address=77.90.31.14:4663 , piece_id=3BXF6WMW4JESZO2LPARD5YPAHR6UFSQLRTK7KVI6UB3SJQUCFBLQ, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.392332220 (49h35m13.43190533s) [noise=true , address=45.91.116.12:4225 , piece_id=PYHW46R3MGBNDVGM76XW2KKJK6SQ2OKVBEUWBBIPRB5DIWTQOY3A, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.392756211 (49h35m13.432329331s) [noise=true , address=77.90.31.19:4843 , piece_id=WHFY2EK32IIC7D3UD5K3HRGFN2O2BAMI4IK4AR7AAOKJC4AXYBFQ, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.393550941 (49h35m13.433124061s) [noise=true , address=77.90.31.18:4806 , piece_id=3WDBC3BXLL23YL4JVDYXQT4OB55WJBOTGDFXSVAIGPDVT7VGV4BA, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.394361923 (49h35m13.433935043s) [noise=true , address=145.79.227.3:4151 , piece_id=EHWQHSRF34YYUFDLMRJVTIYLQJPAYKFUPVPQX3NGLQCDNWHVRBHQ, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.394774653 (49h35m13.434347783s) [noise=true , address=77.90.31.12:4590 , piece_id=FOIQF6523E5GNHKAWSRQIVE4LCFXHNBJNDPSLBTVPKWLCNVKB4KA, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.395162685 (49h35m13.434735805s) [noise=true , address=45.91.116.4:4940 , piece_id=RNCTXSACAALEXR33OOJSQLMAOHG72WSFQCUQHQ6SQ52JPQPACCMQ, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.395376214 (49h35m13.434949324s) [noise=true , address=145.79.227.13:4802 , piece_id=74NHQILYAILBB7X2HJD5C5J35HI7UBCFP6KI6EAZEITUJ5QZ2UJA, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.395603899 (49h35m13.435177019s) [noise=true , address=45.91.116.14:4190 , piece_id=6ZQIFFM2OWZGPNKZ5OD25T2U2LIRIZU335XW2GJNZSKBWGC5PIUQ, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.395814954 (49h35m13.435388074s) [noise=true , address=45.91.116.8:4048 , piece_id=TR4JHZKNRDNJMDHVRT5YVITEKQM53YRZIKCBALB2YNPB7C5FWWQA, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.396008965 (49h35m13.435582085s) [noise=true , address=77.90.31.21:4879 , piece_id=K4LNKLAP6DK56FBO7H2QHZRYPHCZC7JXMMP5KVS6WHTSEEHVYT7Q, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:37.396221762 (49h35m13.435794892s) [noise=true , address=45.91.116.6:4974 , piece_id=QYVBDZBR3RECV3ZQDKXR7M3TUHIINVQ3DILPRW2QULUSCI3LVPHA, seg_pos={0 0}, upload=43572]: Uploading piece... Sep 8 15:03:39.262732068 (49h35m15.302305198s) [upload=43572]: Next returned nil segment Sep 8 15:03:39.262757076 (49h35m15.302330196s) [upload=43572]: Waiting for error group managing segments... Sep 8 15:03:39.563558688 (49h35m15.603131808s) [noise=true , address=77.90.31.14:4663 , piece_id=3BXF6WMW4JESZO2LPARD5YPAHR6UFSQLRTK7KVI6UB3SJQUCFBLQ, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.626060739 (49h35m15.665633859s) [noise=true , address=45.91.116.9:4263 , piece_id=ODM6ZAICRWMXQGW6NCJOFXAX7MZIJTRIRB3LP6QBONZAA74UMG4A, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.667236030 (49h35m15.70680915s) [noise=true , address=77.90.31.21:4879 , piece_id=K4LNKLAP6DK56FBO7H2QHZRYPHCZC7JXMMP5KVS6WHTSEEHVYT7Q, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.684075123 (49h35m15.723648243s) [noise=true , address=77.90.31.19:4841 , piece_id=5AQEHZ5IP2WLCDHXNIY5JQLDLGAEZTRVWAFXQE3GIULYUDMAHYBA, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.733466315 (49h35m15.773039436s) [noise=true , address=45.91.116.4:4940 , piece_id=RNCTXSACAALEXR33OOJSQLMAOHG72WSFQCUQHQ6SQ52JPQPACCMQ, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.733753946 (49h35m15.773327066s) [noise=true , address=45.91.116.10:4083 , piece_id=EPCSBLYDH3D3FNI3Z2R7RO3HZHJ56MYKMUU2EWF3XRCAQXECAZZQ, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.742386394 (49h35m15.781959514s) [noise=true , address=45.91.116.12:4225 , piece_id=PYHW46R3MGBNDVGM76XW2KKJK6SQ2OKVBEUWBBIPRB5DIWTQOY3A, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.742410891 (49h35m15.781984011s) [noise=true , address=77.90.31.19:4843 , piece_id=WHFY2EK32IIC7D3UD5K3HRGFN2O2BAMI4IK4AR7AAOKJC4AXYBFQ, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.758555876 (49h35m15.798129006s) [noise=true , address=45.91.116.14:4190 , piece_id=6ZQIFFM2OWZGPNKZ5OD25T2U2LIRIZU335XW2GJNZSKBWGC5PIUQ, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.769181559 (49h35m15.808754679s) [noise=true , address=77.90.31.12:4590 , piece_id=FOIQF6523E5GNHKAWSRQIVE4LCFXHNBJNDPSLBTVPKWLCNVKB4KA, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.789753624 (49h35m15.829326744s) [noise=true , address=45.91.116.6:4974 , piece_id=QYVBDZBR3RECV3ZQDKXR7M3TUHIINVQ3DILPRW2QULUSCI3LVPHA, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.798309837 (49h35m15.837882957s) [noise=true , address=77.90.31.18:4806 , piece_id=3WDBC3BXLL23YL4JVDYXQT4OB55WJBOTGDFXSVAIGPDVT7VGV4BA, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.859437849 (49h35m15.899010969s) [noise=true , address=145.79.227.11:4730 , piece_id=SWJP3EFCY57ZMESBBTMFO27DDZEP6WEKVITLHC4LRKR6KHHSJB2A, seg_pos={0 0}, upload=43572]: Done uploading piece. err: <nil> Sep 8 15:03:39.859472024 (49h35m15.899045155s) [seg_pos={0 0}, upload=43572]: Segment reached optimal threshold of 13 pieces. Sep 8 15:03:39.859653372 (49h35m15.899226482s) [noise=true , address=45.91.116.8:4048 , piece_id=TR4JHZKNRDNJMDHVRT5YVITEKQM53YRZIKCBALB2YNPB7C5FWWQA, seg_pos={0 0}, upload=43572]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Sep 8 15:03:39.859763182 (49h35m15.899336292s) [noise=true , address=145.79.227.13:4802 , piece_id=74NHQILYAILBB7X2HJD5C5J35HI7UBCFP6KI6EAZEITUJ5QZ2UJA, seg_pos={0 0}, upload=43572]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:54554->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:54554->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54554->145.79.227.13:4802: use of closed network connection Sep 8 15:03:39.859784333 (49h35m15.899357453s) [noise=true , address=145.79.227.3:4151 , piece_id=EHWQHSRF34YYUFDLMRJVTIYLQJPAYKFUPVPQX3NGLQCDNWHVRBHQ, seg_pos={0 0}, upload=43572]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:50582->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:50582->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50582->145.79.227.3:4151: use of closed network connection Sep 8 15:03:39.859802797 (49h35m15.899375917s) [seg_pos={0 0}, upload=43572]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:03:39.859877050 (49h35m15.89945017s) [upload=43572]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:03:39.859880436 (49h35m15.899453556s) [upload=43572]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:03:40.210305304 (49h35m16.249878424s) [upload=43572]: Done uploading segments... Sep 8 15:03:40.210321546 (49h35m16.249894666s) [upload=43572]: Upload finished. err: <nil> Duration: 30.002446028s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XX..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X...] duration: 28.811628892s FAIL (2025-09-08 15:03:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:01:10.316347157 (49h32m46.355920278s) [upload=43537]: Starting upload Sep 8 15:01:10.316413023 (49h32m46.355986143s) [upload=43537]: Done starting upload Sep 8 15:01:10.316516441 (49h32m46.356089561s) [upload=43537]: Uploading segments... Sep 8 15:01:10.316587007 (49h32m46.356160127s) [upload=43537]: Got next segment. Inline: false Sep 8 15:01:10.316613087 (49h32m46.356186207s) [upload=43537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:01:10.316618417 (49h32m46.356191537s) [upload=43537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:01:10.574991718 (49h32m46.614564838s) [seg_pos={0 0}, upload=43537]: Begin upload segment... Sep 8 15:01:10.575045812 (49h32m46.614618922s) [seg_pos={0 0}, upload=43537]: Done begin upload segment. Sep 8 15:01:10.575075899 (49h32m46.614649019s) [noise=true , address=45.91.116.10:4083 , piece_id=6ERU3FAM5XENQP4GU3I7LR3RZBWDIJAP4A6GYUNDX5HLLRSS7NHA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575133711 (49h32m46.614706831s) [noise=true , address=145.79.227.6:4257 , piece_id=OSGR5LMKII4WEF476ULVSYNUVWPNYWO2Q22DGFW2VY4QMU344X5Q, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575547933 (49h32m46.615121063s) [noise=true , address=45.91.116.7:4011 , piece_id=NBCPOPFFAMRH5DKJF44C2T6XE6OQWAGSQQC7ILUHUSCEIFUIUSWA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575672631 (49h32m46.615245761s) [noise=true , address=45.91.116.11:4117 , piece_id=SEGFC7BPSSGV5EYX6YB2KSN3JYAMLREDCNGDJ6G4O2NE2SLCLZMQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576028653 (49h32m46.615601773s) [noise=true , address=45.91.116.7:4010 , piece_id=5BQHOSORHXNRHHFKZYXLHLNPAG2S7VXNGQJJXZTKJ2QHO3QYKOWQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576037901 (49h32m46.615611021s) [noise=true , address=45.91.116.7:4009 , piece_id=WF5X3B2DHGXKDDPFKL4Y5RTQ4GXLLCEVAXAECNVXDMCDHF5MAQMQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576394173 (49h32m46.615967293s) [noise=true , address=77.90.31.10:4517 , piece_id=UG4AXREOOXEQ7DTVVME6ILACSBXQWC3RJAUPD35JZTJN46EJDP6Q, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576444548 (49h32m46.616017668s) [noise=true , address=77.90.31.21:4878 , piece_id=AGY7SQ7PIADHUWBBQ65O4W24ACHS4IKJX6NEC5EBKRVVXYYDDVHQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576773067 (49h32m46.616346187s) [noise=true , address=77.90.31.17:4770 , piece_id=CQSHTBRVAEDKYXU3RN6PQXI3LUSODR5SRYQFCO5HANYBV4A7DAXQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576790310 (49h32m46.61636343s) [noise=true , address=145.79.227.13:4803 , piece_id=EA3E3GBT472JZBIZQIXT744O7O4YU7RLSYDRVBAY65XCEP6LX4PQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577082790 (49h32m46.61665591s) [noise=true , address=45.91.116.12:4227 , piece_id=GVAW5ET4D53D2CQIFF2HKHGSVA4JIKSMDUUW24WNEINQNDUNEACQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577235422 (49h32m46.616808542s) [noise=true , address=77.90.31.18:4805 , piece_id=SGKVI3QO4GTXGBN7MP4TVJSTUNXPTGFNF7XSM4SGLIPAGL3JA5XA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577593788 (49h32m46.617166908s) [noise=true , address=45.91.116.8:4047 , piece_id=LZIXR6JUT3RDQYBAULB7LQH772Z7RLRUHLMNEU446FATZGGQ5C3A, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.579639241 (49h32m46.619212372s) [noise=true , address=77.90.31.13:4627 , piece_id=RCVZHTBTFYCPP2A6JKX4HDJ6VGO4SKT3WTKUIX2DSNB65YAZF5IA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.579988740 (49h32m46.619561861s) [noise=true , address=77.90.31.15:4699 , piece_id=XGGQNDYCPLMU2RCOJ6AGH4JREALM42EBX7LV5B7M75INYZE3NGUA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.580839749 (49h32m46.620412869s) [noise=true , address=145.79.227.4:4186 , piece_id=RJLRLFCYWXLFQGYCKN37LSSTGGTINSMFA55JF76OBPS6VWOVX7LQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.849097877 (49h32m46.888670987s) [noise=true , address=77.90.31.15:4699 , piece_id=XGGQNDYCPLMU2RCOJ6AGH4JREALM42EBX7LV5B7M75INYZE3NGUA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 15:01:12.449678622 (49h32m48.489251742s) [upload=43537]: Next returned nil segment Sep 8 15:01:12.449698160 (49h32m48.48927128s) [upload=43537]: Waiting for error group managing segments... Sep 8 15:01:12.761060950 (49h32m48.80063407s) [noise=true , address=45.91.116.8:4047 , piece_id=LZIXR6JUT3RDQYBAULB7LQH772Z7RLRUHLMNEU446FATZGGQ5C3A, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.773639791 (49h32m48.813212921s) [noise=true , address=45.91.116.10:4083 , piece_id=6ERU3FAM5XENQP4GU3I7LR3RZBWDIJAP4A6GYUNDX5HLLRSS7NHA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.781531893 (49h32m48.821105013s) [noise=true , address=45.91.116.7:4010 , piece_id=5BQHOSORHXNRHHFKZYXLHLNPAG2S7VXNGQJJXZTKJ2QHO3QYKOWQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.800002828 (49h32m48.839575958s) [noise=true , address=45.91.116.11:4117 , piece_id=SEGFC7BPSSGV5EYX6YB2KSN3JYAMLREDCNGDJ6G4O2NE2SLCLZMQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.830010830 (49h32m48.86958396s) [noise=true , address=77.90.31.18:4805 , piece_id=SGKVI3QO4GTXGBN7MP4TVJSTUNXPTGFNF7XSM4SGLIPAGL3JA5XA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.843991543 (49h32m48.883564663s) [noise=true , address=45.91.116.7:4011 , piece_id=NBCPOPFFAMRH5DKJF44C2T6XE6OQWAGSQQC7ILUHUSCEIFUIUSWA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.891061624 (49h32m48.930634754s) [noise=true , address=45.91.116.12:4227 , piece_id=GVAW5ET4D53D2CQIFF2HKHGSVA4JIKSMDUUW24WNEINQNDUNEACQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.950616836 (49h32m48.990189956s) [noise=true , address=77.90.31.10:4517 , piece_id=UG4AXREOOXEQ7DTVVME6ILACSBXQWC3RJAUPD35JZTJN46EJDP6Q, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.121299248 (49h32m49.160872368s) [noise=true , address=77.90.31.21:4878 , piece_id=AGY7SQ7PIADHUWBBQ65O4W24ACHS4IKJX6NEC5EBKRVVXYYDDVHQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.402994538 (49h32m49.442567658s) [noise=true , address=145.79.227.6:4257 , piece_id=OSGR5LMKII4WEF476ULVSYNUVWPNYWO2Q22DGFW2VY4QMU344X5Q, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.436908134 (49h32m49.476481254s) [noise=true , address=145.79.227.4:4186 , piece_id=RJLRLFCYWXLFQGYCKN37LSSTGGTINSMFA55JF76OBPS6VWOVX7LQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.489573821 (49h32m49.529146941s) [noise=true , address=77.90.31.17:4770 , piece_id=CQSHTBRVAEDKYXU3RN6PQXI3LUSODR5SRYQFCO5HANYBV4A7DAXQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.645955358 (49h32m49.685528478s) [noise=true , address=45.91.116.7:4009 , piece_id=WF5X3B2DHGXKDDPFKL4Y5RTQ4GXLLCEVAXAECNVXDMCDHF5MAQMQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.645983261 (49h32m49.685556391s) [seg_pos={0 0}, upload=43537]: Segment reached optimal threshold of 13 pieces. Sep 8 15:01:13.646219733 (49h32m49.685792853s) [noise=true , address=77.90.31.13:4627 , piece_id=RCVZHTBTFYCPP2A6JKX4HDJ6VGO4SKT3WTKUIX2DSNB65YAZF5IA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection Sep 8 15:01:13.646247296 (49h32m49.685820416s) [noise=true , address=145.79.227.13:4803 , piece_id=EA3E3GBT472JZBIZQIXT744O7O4YU7RLSYDRVBAY65XCEP6LX4PQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:01:13.646286581 (49h32m49.685859701s) [seg_pos={0 0}, upload=43537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:01:13.646370963 (49h32m49.685944093s) [upload=43537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:01:13.646375201 (49h32m49.685948321s) [upload=43537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:01:13.983871769 (49h32m50.02344489s) [upload=43537]: Done uploading segments... Sep 8 15:01:13.983891387 (49h32m50.023464507s) [upload=43537]: Upload finished. err: <nil> Duration: 28.811662036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.XX..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X..] duration: 9.25989927s FAIL (2025-09-08 15:02:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:01:10.316347157 (49h32m46.355920278s) [upload=43537]: Starting upload Sep 8 15:01:10.316413023 (49h32m46.355986143s) [upload=43537]: Done starting upload Sep 8 15:01:10.316516441 (49h32m46.356089561s) [upload=43537]: Uploading segments... Sep 8 15:01:10.316587007 (49h32m46.356160127s) [upload=43537]: Got next segment. Inline: false Sep 8 15:01:10.316613087 (49h32m46.356186207s) [upload=43537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:01:10.316618417 (49h32m46.356191537s) [upload=43537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:01:10.574991718 (49h32m46.614564838s) [seg_pos={0 0}, upload=43537]: Begin upload segment... Sep 8 15:01:10.575045812 (49h32m46.614618922s) [seg_pos={0 0}, upload=43537]: Done begin upload segment. Sep 8 15:01:10.575075899 (49h32m46.614649019s) [noise=true , address=45.91.116.10:4083 , piece_id=6ERU3FAM5XENQP4GU3I7LR3RZBWDIJAP4A6GYUNDX5HLLRSS7NHA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575133711 (49h32m46.614706831s) [noise=true , address=145.79.227.6:4257 , piece_id=OSGR5LMKII4WEF476ULVSYNUVWPNYWO2Q22DGFW2VY4QMU344X5Q, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575547933 (49h32m46.615121063s) [noise=true , address=45.91.116.7:4011 , piece_id=NBCPOPFFAMRH5DKJF44C2T6XE6OQWAGSQQC7ILUHUSCEIFUIUSWA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575672631 (49h32m46.615245761s) [noise=true , address=45.91.116.11:4117 , piece_id=SEGFC7BPSSGV5EYX6YB2KSN3JYAMLREDCNGDJ6G4O2NE2SLCLZMQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576028653 (49h32m46.615601773s) [noise=true , address=45.91.116.7:4010 , piece_id=5BQHOSORHXNRHHFKZYXLHLNPAG2S7VXNGQJJXZTKJ2QHO3QYKOWQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576037901 (49h32m46.615611021s) [noise=true , address=45.91.116.7:4009 , piece_id=WF5X3B2DHGXKDDPFKL4Y5RTQ4GXLLCEVAXAECNVXDMCDHF5MAQMQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576394173 (49h32m46.615967293s) [noise=true , address=77.90.31.10:4517 , piece_id=UG4AXREOOXEQ7DTVVME6ILACSBXQWC3RJAUPD35JZTJN46EJDP6Q, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576444548 (49h32m46.616017668s) [noise=true , address=77.90.31.21:4878 , piece_id=AGY7SQ7PIADHUWBBQ65O4W24ACHS4IKJX6NEC5EBKRVVXYYDDVHQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576773067 (49h32m46.616346187s) [noise=true , address=77.90.31.17:4770 , piece_id=CQSHTBRVAEDKYXU3RN6PQXI3LUSODR5SRYQFCO5HANYBV4A7DAXQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576790310 (49h32m46.61636343s) [noise=true , address=145.79.227.13:4803 , piece_id=EA3E3GBT472JZBIZQIXT744O7O4YU7RLSYDRVBAY65XCEP6LX4PQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577082790 (49h32m46.61665591s) [noise=true , address=45.91.116.12:4227 , piece_id=GVAW5ET4D53D2CQIFF2HKHGSVA4JIKSMDUUW24WNEINQNDUNEACQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577235422 (49h32m46.616808542s) [noise=true , address=77.90.31.18:4805 , piece_id=SGKVI3QO4GTXGBN7MP4TVJSTUNXPTGFNF7XSM4SGLIPAGL3JA5XA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577593788 (49h32m46.617166908s) [noise=true , address=45.91.116.8:4047 , piece_id=LZIXR6JUT3RDQYBAULB7LQH772Z7RLRUHLMNEU446FATZGGQ5C3A, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.579639241 (49h32m46.619212372s) [noise=true , address=77.90.31.13:4627 , piece_id=RCVZHTBTFYCPP2A6JKX4HDJ6VGO4SKT3WTKUIX2DSNB65YAZF5IA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.579988740 (49h32m46.619561861s) [noise=true , address=77.90.31.15:4699 , piece_id=XGGQNDYCPLMU2RCOJ6AGH4JREALM42EBX7LV5B7M75INYZE3NGUA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.580839749 (49h32m46.620412869s) [noise=true , address=145.79.227.4:4186 , piece_id=RJLRLFCYWXLFQGYCKN37LSSTGGTINSMFA55JF76OBPS6VWOVX7LQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.849097877 (49h32m46.888670987s) [noise=true , address=77.90.31.15:4699 , piece_id=XGGQNDYCPLMU2RCOJ6AGH4JREALM42EBX7LV5B7M75INYZE3NGUA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 15:01:12.449678622 (49h32m48.489251742s) [upload=43537]: Next returned nil segment Sep 8 15:01:12.449698160 (49h32m48.48927128s) [upload=43537]: Waiting for error group managing segments... Sep 8 15:01:12.761060950 (49h32m48.80063407s) [noise=true , address=45.91.116.8:4047 , piece_id=LZIXR6JUT3RDQYBAULB7LQH772Z7RLRUHLMNEU446FATZGGQ5C3A, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.773639791 (49h32m48.813212921s) [noise=true , address=45.91.116.10:4083 , piece_id=6ERU3FAM5XENQP4GU3I7LR3RZBWDIJAP4A6GYUNDX5HLLRSS7NHA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.781531893 (49h32m48.821105013s) [noise=true , address=45.91.116.7:4010 , piece_id=5BQHOSORHXNRHHFKZYXLHLNPAG2S7VXNGQJJXZTKJ2QHO3QYKOWQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.800002828 (49h32m48.839575958s) [noise=true , address=45.91.116.11:4117 , piece_id=SEGFC7BPSSGV5EYX6YB2KSN3JYAMLREDCNGDJ6G4O2NE2SLCLZMQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.830010830 (49h32m48.86958396s) [noise=true , address=77.90.31.18:4805 , piece_id=SGKVI3QO4GTXGBN7MP4TVJSTUNXPTGFNF7XSM4SGLIPAGL3JA5XA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.843991543 (49h32m48.883564663s) [noise=true , address=45.91.116.7:4011 , piece_id=NBCPOPFFAMRH5DKJF44C2T6XE6OQWAGSQQC7ILUHUSCEIFUIUSWA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.891061624 (49h32m48.930634754s) [noise=true , address=45.91.116.12:4227 , piece_id=GVAW5ET4D53D2CQIFF2HKHGSVA4JIKSMDUUW24WNEINQNDUNEACQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.950616836 (49h32m48.990189956s) [noise=true , address=77.90.31.10:4517 , piece_id=UG4AXREOOXEQ7DTVVME6ILACSBXQWC3RJAUPD35JZTJN46EJDP6Q, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.121299248 (49h32m49.160872368s) [noise=true , address=77.90.31.21:4878 , piece_id=AGY7SQ7PIADHUWBBQ65O4W24ACHS4IKJX6NEC5EBKRVVXYYDDVHQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.402994538 (49h32m49.442567658s) [noise=true , address=145.79.227.6:4257 , piece_id=OSGR5LMKII4WEF476ULVSYNUVWPNYWO2Q22DGFW2VY4QMU344X5Q, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.436908134 (49h32m49.476481254s) [noise=true , address=145.79.227.4:4186 , piece_id=RJLRLFCYWXLFQGYCKN37LSSTGGTINSMFA55JF76OBPS6VWOVX7LQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.489573821 (49h32m49.529146941s) [noise=true , address=77.90.31.17:4770 , piece_id=CQSHTBRVAEDKYXU3RN6PQXI3LUSODR5SRYQFCO5HANYBV4A7DAXQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.645955358 (49h32m49.685528478s) [noise=true , address=45.91.116.7:4009 , piece_id=WF5X3B2DHGXKDDPFKL4Y5RTQ4GXLLCEVAXAECNVXDMCDHF5MAQMQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.645983261 (49h32m49.685556391s) [seg_pos={0 0}, upload=43537]: Segment reached optimal threshold of 13 pieces. Sep 8 15:01:13.646219733 (49h32m49.685792853s) [noise=true , address=77.90.31.13:4627 , piece_id=RCVZHTBTFYCPP2A6JKX4HDJ6VGO4SKT3WTKUIX2DSNB65YAZF5IA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection Sep 8 15:01:13.646247296 (49h32m49.685820416s) [noise=true , address=145.79.227.13:4803 , piece_id=EA3E3GBT472JZBIZQIXT744O7O4YU7RLSYDRVBAY65XCEP6LX4PQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:01:13.646286581 (49h32m49.685859701s) [seg_pos={0 0}, upload=43537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:01:13.646370963 (49h32m49.685944093s) [upload=43537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:01:13.646375201 (49h32m49.685948321s) [upload=43537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:01:13.983871769 (49h32m50.02344489s) [upload=43537]: Done uploading segments... Sep 8 15:01:13.983891387 (49h32m50.023464507s) [upload=43537]: Upload finished. err: <nil> Duration: 9.25993633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X.XX..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X.] duration: 18.602187171s FAIL (2025-09-08 15:01:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:01:10.316347157 (49h32m46.355920278s) [upload=43537]: Starting upload Sep 8 15:01:10.316413023 (49h32m46.355986143s) [upload=43537]: Done starting upload Sep 8 15:01:10.316516441 (49h32m46.356089561s) [upload=43537]: Uploading segments... Sep 8 15:01:10.316587007 (49h32m46.356160127s) [upload=43537]: Got next segment. Inline: false Sep 8 15:01:10.316613087 (49h32m46.356186207s) [upload=43537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:01:10.316618417 (49h32m46.356191537s) [upload=43537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:01:10.574991718 (49h32m46.614564838s) [seg_pos={0 0}, upload=43537]: Begin upload segment... Sep 8 15:01:10.575045812 (49h32m46.614618922s) [seg_pos={0 0}, upload=43537]: Done begin upload segment. Sep 8 15:01:10.575075899 (49h32m46.614649019s) [noise=true , address=45.91.116.10:4083 , piece_id=6ERU3FAM5XENQP4GU3I7LR3RZBWDIJAP4A6GYUNDX5HLLRSS7NHA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575133711 (49h32m46.614706831s) [noise=true , address=145.79.227.6:4257 , piece_id=OSGR5LMKII4WEF476ULVSYNUVWPNYWO2Q22DGFW2VY4QMU344X5Q, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575547933 (49h32m46.615121063s) [noise=true , address=45.91.116.7:4011 , piece_id=NBCPOPFFAMRH5DKJF44C2T6XE6OQWAGSQQC7ILUHUSCEIFUIUSWA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575672631 (49h32m46.615245761s) [noise=true , address=45.91.116.11:4117 , piece_id=SEGFC7BPSSGV5EYX6YB2KSN3JYAMLREDCNGDJ6G4O2NE2SLCLZMQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576028653 (49h32m46.615601773s) [noise=true , address=45.91.116.7:4010 , piece_id=5BQHOSORHXNRHHFKZYXLHLNPAG2S7VXNGQJJXZTKJ2QHO3QYKOWQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576037901 (49h32m46.615611021s) [noise=true , address=45.91.116.7:4009 , piece_id=WF5X3B2DHGXKDDPFKL4Y5RTQ4GXLLCEVAXAECNVXDMCDHF5MAQMQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576394173 (49h32m46.615967293s) [noise=true , address=77.90.31.10:4517 , piece_id=UG4AXREOOXEQ7DTVVME6ILACSBXQWC3RJAUPD35JZTJN46EJDP6Q, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576444548 (49h32m46.616017668s) [noise=true , address=77.90.31.21:4878 , piece_id=AGY7SQ7PIADHUWBBQ65O4W24ACHS4IKJX6NEC5EBKRVVXYYDDVHQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576773067 (49h32m46.616346187s) [noise=true , address=77.90.31.17:4770 , piece_id=CQSHTBRVAEDKYXU3RN6PQXI3LUSODR5SRYQFCO5HANYBV4A7DAXQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576790310 (49h32m46.61636343s) [noise=true , address=145.79.227.13:4803 , piece_id=EA3E3GBT472JZBIZQIXT744O7O4YU7RLSYDRVBAY65XCEP6LX4PQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577082790 (49h32m46.61665591s) [noise=true , address=45.91.116.12:4227 , piece_id=GVAW5ET4D53D2CQIFF2HKHGSVA4JIKSMDUUW24WNEINQNDUNEACQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577235422 (49h32m46.616808542s) [noise=true , address=77.90.31.18:4805 , piece_id=SGKVI3QO4GTXGBN7MP4TVJSTUNXPTGFNF7XSM4SGLIPAGL3JA5XA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577593788 (49h32m46.617166908s) [noise=true , address=45.91.116.8:4047 , piece_id=LZIXR6JUT3RDQYBAULB7LQH772Z7RLRUHLMNEU446FATZGGQ5C3A, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.579639241 (49h32m46.619212372s) [noise=true , address=77.90.31.13:4627 , piece_id=RCVZHTBTFYCPP2A6JKX4HDJ6VGO4SKT3WTKUIX2DSNB65YAZF5IA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.579988740 (49h32m46.619561861s) [noise=true , address=77.90.31.15:4699 , piece_id=XGGQNDYCPLMU2RCOJ6AGH4JREALM42EBX7LV5B7M75INYZE3NGUA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.580839749 (49h32m46.620412869s) [noise=true , address=145.79.227.4:4186 , piece_id=RJLRLFCYWXLFQGYCKN37LSSTGGTINSMFA55JF76OBPS6VWOVX7LQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.849097877 (49h32m46.888670987s) [noise=true , address=77.90.31.15:4699 , piece_id=XGGQNDYCPLMU2RCOJ6AGH4JREALM42EBX7LV5B7M75INYZE3NGUA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 15:01:12.449678622 (49h32m48.489251742s) [upload=43537]: Next returned nil segment Sep 8 15:01:12.449698160 (49h32m48.48927128s) [upload=43537]: Waiting for error group managing segments... Sep 8 15:01:12.761060950 (49h32m48.80063407s) [noise=true , address=45.91.116.8:4047 , piece_id=LZIXR6JUT3RDQYBAULB7LQH772Z7RLRUHLMNEU446FATZGGQ5C3A, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.773639791 (49h32m48.813212921s) [noise=true , address=45.91.116.10:4083 , piece_id=6ERU3FAM5XENQP4GU3I7LR3RZBWDIJAP4A6GYUNDX5HLLRSS7NHA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.781531893 (49h32m48.821105013s) [noise=true , address=45.91.116.7:4010 , piece_id=5BQHOSORHXNRHHFKZYXLHLNPAG2S7VXNGQJJXZTKJ2QHO3QYKOWQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.800002828 (49h32m48.839575958s) [noise=true , address=45.91.116.11:4117 , piece_id=SEGFC7BPSSGV5EYX6YB2KSN3JYAMLREDCNGDJ6G4O2NE2SLCLZMQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.830010830 (49h32m48.86958396s) [noise=true , address=77.90.31.18:4805 , piece_id=SGKVI3QO4GTXGBN7MP4TVJSTUNXPTGFNF7XSM4SGLIPAGL3JA5XA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.843991543 (49h32m48.883564663s) [noise=true , address=45.91.116.7:4011 , piece_id=NBCPOPFFAMRH5DKJF44C2T6XE6OQWAGSQQC7ILUHUSCEIFUIUSWA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.891061624 (49h32m48.930634754s) [noise=true , address=45.91.116.12:4227 , piece_id=GVAW5ET4D53D2CQIFF2HKHGSVA4JIKSMDUUW24WNEINQNDUNEACQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.950616836 (49h32m48.990189956s) [noise=true , address=77.90.31.10:4517 , piece_id=UG4AXREOOXEQ7DTVVME6ILACSBXQWC3RJAUPD35JZTJN46EJDP6Q, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.121299248 (49h32m49.160872368s) [noise=true , address=77.90.31.21:4878 , piece_id=AGY7SQ7PIADHUWBBQ65O4W24ACHS4IKJX6NEC5EBKRVVXYYDDVHQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.402994538 (49h32m49.442567658s) [noise=true , address=145.79.227.6:4257 , piece_id=OSGR5LMKII4WEF476ULVSYNUVWPNYWO2Q22DGFW2VY4QMU344X5Q, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.436908134 (49h32m49.476481254s) [noise=true , address=145.79.227.4:4186 , piece_id=RJLRLFCYWXLFQGYCKN37LSSTGGTINSMFA55JF76OBPS6VWOVX7LQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.489573821 (49h32m49.529146941s) [noise=true , address=77.90.31.17:4770 , piece_id=CQSHTBRVAEDKYXU3RN6PQXI3LUSODR5SRYQFCO5HANYBV4A7DAXQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.645955358 (49h32m49.685528478s) [noise=true , address=45.91.116.7:4009 , piece_id=WF5X3B2DHGXKDDPFKL4Y5RTQ4GXLLCEVAXAECNVXDMCDHF5MAQMQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.645983261 (49h32m49.685556391s) [seg_pos={0 0}, upload=43537]: Segment reached optimal threshold of 13 pieces. Sep 8 15:01:13.646219733 (49h32m49.685792853s) [noise=true , address=77.90.31.13:4627 , piece_id=RCVZHTBTFYCPP2A6JKX4HDJ6VGO4SKT3WTKUIX2DSNB65YAZF5IA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection Sep 8 15:01:13.646247296 (49h32m49.685820416s) [noise=true , address=145.79.227.13:4803 , piece_id=EA3E3GBT472JZBIZQIXT744O7O4YU7RLSYDRVBAY65XCEP6LX4PQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:01:13.646286581 (49h32m49.685859701s) [seg_pos={0 0}, upload=43537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:01:13.646370963 (49h32m49.685944093s) [upload=43537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:01:13.646375201 (49h32m49.685948321s) [upload=43537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:01:13.983871769 (49h32m50.02344489s) [upload=43537]: Done uploading segments... Sep 8 15:01:13.983891387 (49h32m50.023464507s) [upload=43537]: Upload finished. err: <nil> Duration: 18.602208863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..X.XX..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...X] duration: 30.001453085s FAIL (2025-09-08 15:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 15:01:10.316347157 (49h32m46.355920278s) [upload=43537]: Starting upload Sep 8 15:01:10.316413023 (49h32m46.355986143s) [upload=43537]: Done starting upload Sep 8 15:01:10.316516441 (49h32m46.356089561s) [upload=43537]: Uploading segments... Sep 8 15:01:10.316587007 (49h32m46.356160127s) [upload=43537]: Got next segment. Inline: false Sep 8 15:01:10.316613087 (49h32m46.356186207s) [upload=43537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 15:01:10.316618417 (49h32m46.356191537s) [upload=43537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 15:01:10.574991718 (49h32m46.614564838s) [seg_pos={0 0}, upload=43537]: Begin upload segment... Sep 8 15:01:10.575045812 (49h32m46.614618922s) [seg_pos={0 0}, upload=43537]: Done begin upload segment. Sep 8 15:01:10.575075899 (49h32m46.614649019s) [noise=true , address=45.91.116.10:4083 , piece_id=6ERU3FAM5XENQP4GU3I7LR3RZBWDIJAP4A6GYUNDX5HLLRSS7NHA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575133711 (49h32m46.614706831s) [noise=true , address=145.79.227.6:4257 , piece_id=OSGR5LMKII4WEF476ULVSYNUVWPNYWO2Q22DGFW2VY4QMU344X5Q, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575547933 (49h32m46.615121063s) [noise=true , address=45.91.116.7:4011 , piece_id=NBCPOPFFAMRH5DKJF44C2T6XE6OQWAGSQQC7ILUHUSCEIFUIUSWA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.575672631 (49h32m46.615245761s) [noise=true , address=45.91.116.11:4117 , piece_id=SEGFC7BPSSGV5EYX6YB2KSN3JYAMLREDCNGDJ6G4O2NE2SLCLZMQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576028653 (49h32m46.615601773s) [noise=true , address=45.91.116.7:4010 , piece_id=5BQHOSORHXNRHHFKZYXLHLNPAG2S7VXNGQJJXZTKJ2QHO3QYKOWQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576037901 (49h32m46.615611021s) [noise=true , address=45.91.116.7:4009 , piece_id=WF5X3B2DHGXKDDPFKL4Y5RTQ4GXLLCEVAXAECNVXDMCDHF5MAQMQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576394173 (49h32m46.615967293s) [noise=true , address=77.90.31.10:4517 , piece_id=UG4AXREOOXEQ7DTVVME6ILACSBXQWC3RJAUPD35JZTJN46EJDP6Q, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576444548 (49h32m46.616017668s) [noise=true , address=77.90.31.21:4878 , piece_id=AGY7SQ7PIADHUWBBQ65O4W24ACHS4IKJX6NEC5EBKRVVXYYDDVHQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576773067 (49h32m46.616346187s) [noise=true , address=77.90.31.17:4770 , piece_id=CQSHTBRVAEDKYXU3RN6PQXI3LUSODR5SRYQFCO5HANYBV4A7DAXQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.576790310 (49h32m46.61636343s) [noise=true , address=145.79.227.13:4803 , piece_id=EA3E3GBT472JZBIZQIXT744O7O4YU7RLSYDRVBAY65XCEP6LX4PQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577082790 (49h32m46.61665591s) [noise=true , address=45.91.116.12:4227 , piece_id=GVAW5ET4D53D2CQIFF2HKHGSVA4JIKSMDUUW24WNEINQNDUNEACQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577235422 (49h32m46.616808542s) [noise=true , address=77.90.31.18:4805 , piece_id=SGKVI3QO4GTXGBN7MP4TVJSTUNXPTGFNF7XSM4SGLIPAGL3JA5XA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.577593788 (49h32m46.617166908s) [noise=true , address=45.91.116.8:4047 , piece_id=LZIXR6JUT3RDQYBAULB7LQH772Z7RLRUHLMNEU446FATZGGQ5C3A, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.579639241 (49h32m46.619212372s) [noise=true , address=77.90.31.13:4627 , piece_id=RCVZHTBTFYCPP2A6JKX4HDJ6VGO4SKT3WTKUIX2DSNB65YAZF5IA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.579988740 (49h32m46.619561861s) [noise=true , address=77.90.31.15:4699 , piece_id=XGGQNDYCPLMU2RCOJ6AGH4JREALM42EBX7LV5B7M75INYZE3NGUA, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.580839749 (49h32m46.620412869s) [noise=true , address=145.79.227.4:4186 , piece_id=RJLRLFCYWXLFQGYCKN37LSSTGGTINSMFA55JF76OBPS6VWOVX7LQ, seg_pos={0 0}, upload=43537]: Uploading piece... Sep 8 15:01:10.849097877 (49h32m46.888670987s) [noise=true , address=77.90.31.15:4699 , piece_id=XGGQNDYCPLMU2RCOJ6AGH4JREALM42EBX7LV5B7M75INYZE3NGUA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 15:01:12.449678622 (49h32m48.489251742s) [upload=43537]: Next returned nil segment Sep 8 15:01:12.449698160 (49h32m48.48927128s) [upload=43537]: Waiting for error group managing segments... Sep 8 15:01:12.761060950 (49h32m48.80063407s) [noise=true , address=45.91.116.8:4047 , piece_id=LZIXR6JUT3RDQYBAULB7LQH772Z7RLRUHLMNEU446FATZGGQ5C3A, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.773639791 (49h32m48.813212921s) [noise=true , address=45.91.116.10:4083 , piece_id=6ERU3FAM5XENQP4GU3I7LR3RZBWDIJAP4A6GYUNDX5HLLRSS7NHA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.781531893 (49h32m48.821105013s) [noise=true , address=45.91.116.7:4010 , piece_id=5BQHOSORHXNRHHFKZYXLHLNPAG2S7VXNGQJJXZTKJ2QHO3QYKOWQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.800002828 (49h32m48.839575958s) [noise=true , address=45.91.116.11:4117 , piece_id=SEGFC7BPSSGV5EYX6YB2KSN3JYAMLREDCNGDJ6G4O2NE2SLCLZMQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.830010830 (49h32m48.86958396s) [noise=true , address=77.90.31.18:4805 , piece_id=SGKVI3QO4GTXGBN7MP4TVJSTUNXPTGFNF7XSM4SGLIPAGL3JA5XA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.843991543 (49h32m48.883564663s) [noise=true , address=45.91.116.7:4011 , piece_id=NBCPOPFFAMRH5DKJF44C2T6XE6OQWAGSQQC7ILUHUSCEIFUIUSWA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.891061624 (49h32m48.930634754s) [noise=true , address=45.91.116.12:4227 , piece_id=GVAW5ET4D53D2CQIFF2HKHGSVA4JIKSMDUUW24WNEINQNDUNEACQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:12.950616836 (49h32m48.990189956s) [noise=true , address=77.90.31.10:4517 , piece_id=UG4AXREOOXEQ7DTVVME6ILACSBXQWC3RJAUPD35JZTJN46EJDP6Q, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.121299248 (49h32m49.160872368s) [noise=true , address=77.90.31.21:4878 , piece_id=AGY7SQ7PIADHUWBBQ65O4W24ACHS4IKJX6NEC5EBKRVVXYYDDVHQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.402994538 (49h32m49.442567658s) [noise=true , address=145.79.227.6:4257 , piece_id=OSGR5LMKII4WEF476ULVSYNUVWPNYWO2Q22DGFW2VY4QMU344X5Q, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.436908134 (49h32m49.476481254s) [noise=true , address=145.79.227.4:4186 , piece_id=RJLRLFCYWXLFQGYCKN37LSSTGGTINSMFA55JF76OBPS6VWOVX7LQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.489573821 (49h32m49.529146941s) [noise=true , address=77.90.31.17:4770 , piece_id=CQSHTBRVAEDKYXU3RN6PQXI3LUSODR5SRYQFCO5HANYBV4A7DAXQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.645955358 (49h32m49.685528478s) [noise=true , address=45.91.116.7:4009 , piece_id=WF5X3B2DHGXKDDPFKL4Y5RTQ4GXLLCEVAXAECNVXDMCDHF5MAQMQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: <nil> Sep 8 15:01:13.645983261 (49h32m49.685556391s) [seg_pos={0 0}, upload=43537]: Segment reached optimal threshold of 13 pieces. Sep 8 15:01:13.646219733 (49h32m49.685792853s) [noise=true , address=77.90.31.13:4627 , piece_id=RCVZHTBTFYCPP2A6JKX4HDJ6VGO4SKT3WTKUIX2DSNB65YAZF5IA, seg_pos={0 0}, upload=43537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39342->77.90.31.13:4627: use of closed network connection Sep 8 15:01:13.646247296 (49h32m49.685820416s) [noise=true , address=145.79.227.13:4803 , piece_id=EA3E3GBT472JZBIZQIXT744O7O4YU7RLSYDRVBAY65XCEP6LX4PQ, seg_pos={0 0}, upload=43537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Sep 8 15:01:13.646286581 (49h32m49.685859701s) [seg_pos={0 0}, upload=43537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 15:01:13.646370963 (49h32m49.685944093s) [upload=43537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 15:01:13.646375201 (49h32m49.685948321s) [upload=43537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 15:01:13.983871769 (49h32m50.02344489s) [upload=43537]: Done uploading segments... Sep 8 15:01:13.983891387 (49h32m50.023464507s) [upload=43537]: Upload finished. err: <nil> Duration: 30.001473064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..X.XX..X.X........XXX.......XXX.XXXXXX.......X.XX..XXX...] duration: 22.851038644s FAIL (2025-09-08 15:00:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 14:57:50.482907219 (49h29m26.522480339s) [upload=43488]: Starting upload Sep 8 14:57:50.483020517 (49h29m26.522593637s) [upload=43488]: Done starting upload Sep 8 14:57:50.483093686 (49h29m26.522666796s) [upload=43488]: Uploading segments... Sep 8 14:57:50.483237623 (49h29m26.522810743s) [upload=43488]: Got next segment. Inline: false Sep 8 14:57:50.483273130 (49h29m26.52284625s) [upload=43488]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 14:57:50.483277468 (49h29m26.522850588s) [upload=43488]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 14:57:51.042125567 (49h29m27.081698677s) [seg_pos={0 0}, upload=43488]: Begin upload segment... Sep 8 14:57:51.042179740 (49h29m27.08175289s) [seg_pos={0 0}, upload=43488]: Done begin upload segment. Sep 8 14:57:51.042218375 (49h29m27.081791495s) [noise=true , address=77.90.31.10:4520 , piece_id=UQND4CFLW4TNCIT4JEIFX65IEJ5QGLCEHLTSCZ3ZOJKBGEQB52LA, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.042237320 (49h29m27.08181044s) [noise=true , address=77.90.31.15:4698 , piece_id=AO66OJL4DCQ3SCCDMMISBIYL7WINFKO4N33VXBBEANJFZUZKWMVA, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.042586258 (49h29m27.082159378s) [noise=true , address=145.79.227.14:4839 , piece_id=CHU5USPYWWDZMOI4PGEVIMKDQKAVLGPQSWICW4MH2326ICSOA6SQ, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.042674026 (49h29m27.082247146s) [noise=true , address=45.91.116.9:4261 , piece_id=4KMGS3WR7TJSG775V6SFY7CB43TGNBL2M3HIRERCNULGR72DXICQ, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.042908415 (49h29m27.082481535s) [noise=true , address=77.90.31.21:4878 , piece_id=GJXAP3FAGCTBYRFVELKTNBEGXRYL2LQHYZGXDWTZCDV4QIRXYYHA, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.043192608 (49h29m27.082765758s) [noise=true , address=45.91.116.13:4155 , piece_id=WNJLGQN52BWJYV74EIJY3ZDCOISBUBLX2Y6INIMSHNBVPOAYJQMQ, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.043486391 (49h29m27.083059511s) [noise=true , address=77.90.31.15:4697 , piece_id=3KKRON53NTVTHLWJHGTGDQB4MQBVAZL5HOS2O5RALUF4KWOF3JSA, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.044305498 (49h29m27.083878608s) [noise=true , address=45.91.116.6:4974 , piece_id=YQMCHJK3SE4Q7A3QNL5FSQW5TRUR2KJIRWCLTASCIVNHSEUO6R5A, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.044589451 (49h29m27.084162571s) [noise=true , address=45.91.116.5:4299 , piece_id=5YKAOILPFRUL37RFXMSAMEWFGDU2LTNQNCHMD6FDPGKWOAQ7KPGQ, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.044869738 (49h29m27.084442868s) [noise=true , address=77.90.31.14:4663 , piece_id=5EJ2V3BUMRTZOXXZHEGMAJN3BY5IIN5D3LAAE72WMSCCF54ADHSA, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.045109707 (49h29m27.084682827s) [noise=true , address=45.91.116.5:4298 , piece_id=GIJGMT7SIIPOFFUJHGV2QBQH5XOBYMWWUMJ3PVDQS6MATNCGOJOQ, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.045230098 (49h29m27.084803218s) [noise=true , address=145.79.227.5:4223 , piece_id=LIILEFHOMIXBMN2H5QXHWIBL22HZ53ZMQUFPDAQUEN67W26S3ZCA, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.045537736 (49h29m27.085110856s) [noise=true , address=45.91.116.8:4048 , piece_id=4TXGLHSZPP3TXLXJWMVDR4ME7ZUACYXDBJCPD7REAJ7SU62Z6ZJA, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.045837239 (49h29m27.085410359s) [noise=true , address=77.90.31.21:4879 , piece_id=ZRBC2646TULJFRDKU54SOAQAFHE3SH4S5H6RSPDMN2KPJXVQOV4Q, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.046091004 (49h29m27.085664124s) [noise=true , address=45.91.116.8:4045 , piece_id=27U6Q73YBS7LKZKT7VHQABPNLXCD3HGMA5SBCREC4VXN4C5XLTAA, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.046317438 (49h29m27.085890558s) [noise=true , address=45.91.116.14:4191 , piece_id=5PTY2YJGETWDE5UPLVDOJUFJWZ4BRIV4U7HQBPM6YZ3DSPW6YYWQ, seg_pos={0 0}, upload=43488]: Uploading piece... Sep 8 14:57:51.320325599 (49h29m27.359898709s) [noise=true , address=77.90.31.15:4697 , piece_id=3KKRON53NTVTHLWJHGTGDQB4MQBVAZL5HOS2O5RALUF4KWOF3JSA, seg_pos={0 0}, upload=43488]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:56524->77.90.31.15:4697: write: connection refused; manager closed: conn closed: op Sep 8 14:57:52.938006134 (49h29m28.977579254s) [upload=43488]: Next returned nil segment Sep 8 14:57:52.938023418 (49h29m28.977596538s) [upload=43488]: Waiting for error group managing segments... Sep 8 14:57:53.233729699 (49h29m29.273302839s) [noise=true , address=77.90.31.14:4663 , piece_id=5EJ2V3BUMRTZOXXZHEGMAJN3BY5IIN5D3LAAE72WMSCCF54ADHSA, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.238089751 (49h29m29.277662872s) [noise=true , address=77.90.31.15:4698 , piece_id=AO66OJL4DCQ3SCCDMMISBIYL7WINFKO4N33VXBBEANJFZUZKWMVA, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.242402825 (49h29m29.281975975s) [noise=true , address=45.91.116.9:4261 , piece_id=4KMGS3WR7TJSG775V6SFY7CB43TGNBL2M3HIRERCNULGR72DXICQ, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.263395045 (49h29m29.302968165s) [noise=true , address=77.90.31.21:4879 , piece_id=ZRBC2646TULJFRDKU54SOAQAFHE3SH4S5H6RSPDMN2KPJXVQOV4Q, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.278756350 (49h29m29.31832947s) [noise=true , address=45.91.116.5:4298 , piece_id=GIJGMT7SIIPOFFUJHGV2QBQH5XOBYMWWUMJ3PVDQS6MATNCGOJOQ, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.282093085 (49h29m29.321666205s) [noise=true , address=45.91.116.5:4299 , piece_id=5YKAOILPFRUL37RFXMSAMEWFGDU2LTNQNCHMD6FDPGKWOAQ7KPGQ, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.328035679 (49h29m29.367608799s) [noise=true , address=77.90.31.21:4878 , piece_id=GJXAP3FAGCTBYRFVELKTNBEGXRYL2LQHYZGXDWTZCDV4QIRXYYHA, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.362233298 (49h29m29.401806418s) [noise=true , address=45.91.116.8:4048 , piece_id=4TXGLHSZPP3TXLXJWMVDR4ME7ZUACYXDBJCPD7REAJ7SU62Z6ZJA, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.369613591 (49h29m29.409186701s) [noise=true , address=45.91.116.14:4191 , piece_id=5PTY2YJGETWDE5UPLVDOJUFJWZ4BRIV4U7HQBPM6YZ3DSPW6YYWQ, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.388474151 (49h29m29.428047271s) [noise=true , address=77.90.31.10:4520 , piece_id=UQND4CFLW4TNCIT4JEIFX65IEJ5QGLCEHLTSCZ3ZOJKBGEQB52LA, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.414941548 (49h29m29.454514668s) [noise=true , address=45.91.116.6:4974 , piece_id=YQMCHJK3SE4Q7A3QNL5FSQW5TRUR2KJIRWCLTASCIVNHSEUO6R5A, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.556969797 (49h29m29.596542917s) [noise=true , address=45.91.116.13:4155 , piece_id=WNJLGQN52BWJYV74EIJY3ZDCOISBUBLX2Y6INIMSHNBVPOAYJQMQ, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.563541854 (49h29m29.603114974s) [noise=true , address=45.91.116.8:4045 , piece_id=27U6Q73YBS7LKZKT7VHQABPNLXCD3HGMA5SBCREC4VXN4C5XLTAA, seg_pos={0 0}, upload=43488]: Done uploading piece. err: <nil> Sep 8 14:57:53.563552495 (49h29m29.603125615s) [seg_pos={0 0}, upload=43488]: Segment reached optimal threshold of 13 pieces. Sep 8 14:57:53.563706108 (49h29m29.603279218s) [noise=true , address=145.79.227.5:4223 , piece_id=LIILEFHOMIXBMN2H5QXHWIBL22HZ53ZMQUFPDAQUEN67W26S3ZCA, seg_pos={0 0}, upload=43488]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: expected piece hash; context canceled; EOF Sep 8 14:57:53.563719704 (49h29m29.603292824s) [noise=true , address=145.79.227.14:4839 , piece_id=CHU5USPYWWDZMOI4PGEVIMKDQKAVLGPQSWICW4MH2326ICSOA6SQ, seg_pos={0 0}, upload=43488]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:56768->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:56768->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56768->145.79.227.14:4839: use of closed network connection Sep 8 14:57:53.563740543 (49h29m29.603313664s) [seg_pos={0 0}, upload=43488]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 14:57:53.563806931 (49h29m29.603380051s) [upload=43488]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 14:57:53.563811059 (49h29m29.603384179s) [upload=43488]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 14:57:53.960273764 (49h29m29.999846884s) [upload=43488]: Done uploading segments... Sep 8 14:57:53.960321585 (49h29m29.999894695s) [upload=43488]: Upload finished. err: <nil> Duration: 22.851066707s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)