Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.14911127s FAIL (2025-09-27 22:21:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.149140904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.011766056s FAIL (2025-09-27 22:20:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.011836509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.205607047s FAIL (2025-09-27 22:20:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.205641593s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.220988788s FAIL (2025-09-27 22:19:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.221019616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.2243144s FAIL (2025-09-27 22:18:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.224345931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.196411217s FAIL (2025-09-27 22:17:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.196443648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.917657248s FAIL (2025-09-27 22:17:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.917694469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.15226772s FAIL (2025-09-27 22:17:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.152306052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.292134327s FAIL (2025-09-27 22:16:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.292163933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.572157242s FAIL (2025-09-27 22:15:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.572196868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.441058749s FAIL (2025-09-27 22:14:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.441091512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.015629028s FAIL (2025-09-27 22:14:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.01566148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.106170971s FAIL (2025-09-27 22:13:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.106203404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.054176965s FAIL (2025-09-27 22:13:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.054212322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.126080777s FAIL (2025-09-27 22:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.126112968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 22.293042685s FAIL (2025-09-27 22:11:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 22.293071789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.218279287s FAIL (2025-09-27 22:10:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.218315064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.266487463s FAIL (2025-09-27 22:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 22:09:33.954308444 (104h38m19.416400917s) [upload=92107]: Starting upload Sep 27 22:09:33.954377695 (104h38m19.416470169s) [upload=92107]: Done starting upload Sep 27 22:09:33.954534795 (104h38m19.416627278s) [upload=92107]: Uploading segments... Sep 27 22:09:33.954622191 (104h38m19.416714664s) [upload=92107]: Got next segment. Inline: false Sep 27 22:09:33.954659954 (104h38m19.416752437s) [upload=92107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 22:09:33.954666356 (104h38m19.416758839s) [upload=92107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 22:09:34.213355347 (104h38m19.67544783s) [seg_pos={0 0}, upload=92107]: Begin upload segment... Sep 27 22:09:34.213418917 (104h38m19.67551139s) [seg_pos={0 0}, upload=92107]: Done begin upload segment. Sep 27 22:09:34.213446831 (104h38m19.675539304s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213481467 (104h38m19.67557394s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.213934169 (104h38m19.676026642s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.214199394 (104h38m19.676291867s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215100821 (104h38m19.677193304s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215429496 (104h38m19.677521969s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.215842152 (104h38m19.677934625s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.216217607 (104h38m19.67831009s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217331658 (104h38m19.679424131s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.217693778 (104h38m19.679786251s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.218851142 (104h38m19.680943615s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.219444412 (104h38m19.681536895s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.221118880 (104h38m19.683211363s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.222465464 (104h38m19.684557947s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.223818921 (104h38m19.685911404s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:34.225250336 (104h38m19.687342809s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Uploading piece... Sep 27 22:09:35.952157363 (104h38m21.414249836s) [upload=92107]: Next returned nil segment Sep 27 22:09:35.952177822 (104h38m21.414270295s) [upload=92107]: Waiting for error group managing segments... Sep 27 22:09:36.237714787 (104h38m21.69980726s) [noise=true , address=145.79.227.4:4187 , piece_id=MB4XNSKNLLS5VG6XZPKHSG4OPQMIZWEJVHMF6WG2Z3PZJGGVX3HQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.241311916 (104h38m21.703404389s) [noise=true , address=145.79.227.13:4801 , piece_id=MJMYKCBDOKOQ7ZSZQWRZ6MV3PAXT77MVNMLPIPPAIMOEL6RJR5EA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249032137 (104h38m21.71112461s) [noise=true , address=145.79.227.17:4969 , piece_id=RMVH54NM4VAUTCU3JMBDVC76ZNL6HHNC7OWQL73O44PNBIBRIRGA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.249316468 (104h38m21.711408951s) [noise=true , address=145.79.227.5:4223 , piece_id=B3MH7FERM2ZKSLY24VK7OOVIP4OIXRGC4PCOYMEXZK6O2Z4UNR7A, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.252613236 (104h38m21.714705709s) [noise=true , address=145.79.227.19:4079 , piece_id=LBUB722I77BM3CMBL3NVKGPNROM7SAKOXOSXIEHIZLGNZSG2VC4Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.256940876 (104h38m21.719033359s) [noise=true , address=145.79.227.16:4910 , piece_id=BEABRVIFC5PSLTV234FP7BDX7W2MBV57E67ZB2UQ4T45Z5GKTB2Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.288963312 (104h38m21.751055785s) [noise=true , address=77.90.31.11:4553 , piece_id=TBW45DMVPYONVN7TOTK6N5SKKXCY6BHNM4MAWECC42Q53Y5EDMMA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.331203976 (104h38m21.79329646s) [noise=true , address=77.90.31.15:4697 , piece_id=3AQ5K67LV4JKINWJL5NB6K5YWJD3DVB63FFMOCLTP5O337YSHKAQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.417568651 (104h38m21.879661134s) [noise=true , address=77.90.31.21:4877 , piece_id=3VQTLHVDZ236ZFA2YFXMOMJND6CONRQTSV7GLHU4GHH3SBODMSIA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.887025212 (104h38m22.349117685s) [noise=true , address=77.90.31.14:4662 , piece_id=F5STWWG5XS3QQBMZRPVIEZOI53DAKWPRN5MVWDHDOMRRFGYB266Q, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:09:36.998068038 (104h38m22.460160521s) [noise=true , address=77.90.31.12:4590 , piece_id=DMPIWVQMUGSYG4KQ6LWXIDWLU3X4REHLRN65TSB5TEJMP6TNYWDA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: <nil> Sep 27 22:10:03.229770833 (104h38m48.691863316s) [noise=true , address=45.91.116.7:4010 , piece_id=2U7F6NPCBVNLXIE4LMTIYKMFR4MOUFVXZZGBZDABDIUAI67ZLHSQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:52932->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230459585 (104h38m48.692552068s) [noise=true , address=45.91.116.4:4937 , piece_id=C6FZEZ35C3GONJLLTZWQMIEFNLUHZ5MWSDSFVWXFHDDRWBEHMSEQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:39106->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230615221 (104h38m48.692707714s) [noise=true , address=45.91.116.14:4191 , piece_id=JQC6HYEWTZEXXLLFZGNPHY6B4ZJLQTULCH7R7YJSNC5HT4NHACVQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:51846->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.230745870 (104h38m48.692838353s) [noise=true , address=45.91.116.10:4081 , piece_id=VAHDD3KMK57PW5RTKSG7WCQABZ2X2GBSJRTZQ3VL3VB4KJDFU3XA, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:50430->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 22:10:03.231003070 (104h38m48.693095553s) [noise=true , address=45.91.116.9:4262 , piece_id=AI2HPRFG4RPZ3EZA3QKR33RYD43NXETH5ULBH6FBGKSUWFJHNGCQ, seg_pos={0 0}, upload=92107]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:43668->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 22:10:03.231046112 (104h38m48.693138595s) [seg_pos={0 0}, upload=92107]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 22:10:03.493884075 (104h38m48.955976558s) [upload=92107]: Done uploading segments... Sep 27 22:10:03.493924192 (104h38m48.956016675s) [upload=92107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.266503513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.071788312s FAIL (2025-09-27 21:45:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.071821394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.085304237s FAIL (2025-09-27 21:44:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.08533235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.851652194s FAIL (2025-09-27 21:44:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 5.851683422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.413120341s FAIL (2025-09-27 21:43:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.413151391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 6.259370825s FAIL (2025-09-27 21:43:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.259416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 6.089918073s FAIL (2025-09-27 21:42:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.089954333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 5.976571041s FAIL (2025-09-27 21:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 5.97659209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 6.184925182s FAIL (2025-09-27 21:41:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.184956532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 6.012907429s FAIL (2025-09-27 21:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.012940091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration: 6.275086599s FAIL (2025-09-27 21:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.275116344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 6.062348936s FAIL (2025-09-27 21:38:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.06237748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 6.101920749s FAIL (2025-09-27 21:37:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.102053581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 10.965659809s FAIL (2025-09-27 21:36:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 10.965693042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 6.090738205s FAIL (2025-09-27 21:35:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.090766348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 6.183476928s FAIL (2025-09-27 21:34:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.183509339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 6.175662055s FAIL (2025-09-27 21:33:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.175696741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 6.033258201s FAIL (2025-09-27 21:33:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.0332945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 6.094640848s FAIL (2025-09-27 21:32:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.094667628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 5.926755099s FAIL (2025-09-27 21:31:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 5.926788503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 6.101386709s FAIL (2025-09-27 21:31:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.101422396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 5.968326019s FAIL (2025-09-27 21:30:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 5.968355497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 5.777431827s FAIL (2025-09-27 21:29:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 5.777466323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 6.252365603s FAIL (2025-09-27 21:29:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.25245363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 6.131489415s FAIL (2025-09-27 21:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.131521205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 6.169445899s FAIL (2025-09-27 21:27:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.169473942s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 6.319669126s FAIL (2025-09-27 21:26:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.319696349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 6.161877962s FAIL (2025-09-27 21:26:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.161911455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X......................X...X................................] duration: 6.171241708s FAIL (2025-09-27 21:25:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.171276223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X......................X...X...............................] duration: 6.0348805s FAIL (2025-09-27 21:24:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.034911821s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X......................X...X..............................] duration: 6.103292029s FAIL (2025-09-27 21:23:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.103324321s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X......................X...X.............................] duration: 6.05083035s FAIL (2025-09-27 21:22:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.050867972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X......................X...X............................] duration: 6.271531318s FAIL (2025-09-27 21:22:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 21:01:03.445451159 (103h29m48.907543652s) [upload=91097]: Starting upload Sep 27 21:01:03.445539026 (103h29m48.907631509s) [upload=91097]: Done starting upload Sep 27 21:01:03.445739177 (103h29m48.90783167s) [upload=91097]: Uploading segments... Sep 27 21:01:03.445908730 (103h29m48.908001233s) [upload=91097]: Got next segment. Inline: false Sep 27 21:01:03.445965799 (103h29m48.908058282s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 21:01:03.445975868 (103h29m48.908068351s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 21:01:03.697543092 (103h29m49.159635595s) [seg_pos={0 0}, upload=91097]: Begin upload segment... Sep 27 21:01:03.697630958 (103h29m49.159723431s) [seg_pos={0 0}, upload=91097]: Done begin upload segment. Sep 27 21:01:03.697671496 (103h29m49.15976398s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.697770705 (103h29m49.159863188s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698389733 (103h29m49.160482217s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.698795176 (103h29m49.160887659s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699263748 (103h29m49.161356231s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.699695721 (103h29m49.161788204s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700181666 (103h29m49.162274149s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.700840280 (103h29m49.162932763s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.701352415 (103h29m49.163444898s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.702902797 (103h29m49.164995281s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.704122490 (103h29m49.166214973s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.708305355 (103h29m49.170397838s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.711318372 (103h29m49.173410856s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.714979293 (103h29m49.177071806s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.716341307 (103h29m49.17843379s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:03.726091132 (103h29m49.188183615s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Uploading piece... Sep 27 21:01:05.451342684 (103h29m50.913435167s) [upload=91097]: Next returned nil segment Sep 27 21:01:05.451398971 (103h29m50.913491454s) [upload=91097]: Waiting for error group managing segments... Sep 27 21:01:05.738438151 (103h29m51.200530634s) [noise=true , address=145.79.227.11:4731 , piece_id=K3YTLHGTDUEX6YYY4YKQ52DF7POW7RNRSX75YKAN7VM5DCEUCE7Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.743285371 (103h29m51.205377845s) [noise=true , address=145.79.227.4:4187 , piece_id=IDOBISTK2SIOX5OH5TXYJR3EZVBU4HT6GE7FKT42YULQBNGE3JNQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.747176751 (103h29m51.209269234s) [noise=true , address=145.79.227.14:4839 , piece_id=HFIDE63IKXOMWZ3GQUIKLBSIBDYTTTPG2T7VDF7GZJCQIIBS6J2A, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.748578370 (103h29m51.210670853s) [noise=true , address=145.79.227.6:4257 , piece_id=ALYVYX2G3GE6HEKKZVJLEXUA3KHTCMNRRPPYBOZFMEEZRK5YXLCQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.758178240 (103h29m51.220270723s) [noise=true , address=145.79.227.2:4114 , piece_id=7KTY4MXUIK2XYTH727PIXAGNPRLWUJSGXRSKJUIJRW76MOJJB3EA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.770996278 (103h29m51.233088761s) [noise=true , address=145.79.227.5:4221 , piece_id=N5BRIQYXP3ANVC3A2AWAODTGGUBOVDE5CKET6VSHYQJMLT4HE54Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:05.796128034 (103h29m51.258220517s) [noise=true , address=145.79.227.15:4876 , piece_id=3TGZ3TRHK56O5TGTGEU5ISVNJN2AATUIUJIPXUDN3P7EBO2RPFDQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.107162996 (103h29m51.569255479s) [noise=true , address=45.91.116.8:4045 , piece_id=BOKGQR7HCPHF7XCOLXN52ZOKNWQASWEFSPZN62FZWXXS6FGI3Q6Q, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.133884490 (103h29m51.595976973s) [noise=true , address=45.91.116.14:4190 , piece_id=CFVXI6UKKZWWSB6B5HB3ZIIM7GINUDFYJXIHAXVYVRJ3MBNP7MPA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.526257516 (103h29m51.988349989s) [noise=true , address=45.91.116.6:4975 , piece_id=FYMT2OMP3B6PBY5TUXAHKTI4HELJPZWBVIJC4IVIMMWL6O2KYVJA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:06.599082650 (103h29m52.061175133s) [noise=true , address=45.91.116.9:4261 , piece_id=6ABSLD6HDN67WKQUOVSICOR5TMRBSGE77OMD2NGMET5HZ5KX34WA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:07.953408105 (103h29m53.415500588s) [noise=true , address=145.79.227.16:4909 , piece_id=V22SXOUZSYBKTFY7FXLDJOP5U6XNBXWNP7NNHKSR2HHIWG6PVJWA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987709885 (103h30m7.449802369s) [noise=true , address=77.90.31.11:4555 , piece_id=JZPFL6U7CVAMDIFDUM6LQ64BLGX52T64DHI5O7HK5LFT2LP234FQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: <nil> Sep 27 21:01:21.987944732 (103h30m7.450037225s) [seg_pos={0 0}, upload=91097]: Segment reached optimal threshold of 13 pieces. Sep 27 21:01:21.988127360 (103h30m7.450219844s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988139193 (103h30m7.450231676s) [noise=true , address=77.90.31.10:4520 , piece_id=QJUKYV6PRS6N4NA7UXX3L2N3OAAG3UNTSJVC56QQWISYGMW2DIQQ, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988170452 (103h30m7.450262935s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988180691 (103h30m7.450273175s) [noise=true , address=77.90.31.19:4842 , piece_id=57XBWGFBWUZZ2AKVYQ6MPHPLBUX4YADNAVBWEIZIEPCJPHNCV5VA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988252648 (103h30m7.450345131s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Sep 27 21:01:21.988261366 (103h30m7.450353849s) [noise=true , address=77.90.31.12:4589 , piece_id=AHVG4BMWBYAIYQWVJWUGXQSM5BPB34KTWIQUJ63OCVL4IIPIFOUA, seg_pos={0 0}, upload=91097]: Long tail cancellation due to optimal threshold Sep 27 21:01:21.988279650 (103h30m7.450372133s) [seg_pos={0 0}, upload=91097]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 21:01:21.988366847 (103h30m7.45045932s) [upload=91097]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 21:01:21.988374421 (103h30m7.450466904s) [upload=91097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 21:01:22.294530513 (103h30m7.756623046s) [upload=91097]: Done uploading segments... Sep 27 21:01:22.294551582 (103h30m7.756644066s) [upload=91097]: Upload finished. err: <nil> Duration: 6.271560735s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)