Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................................XXX.X..X.....XX] duration: 30.000945318s FAIL (2025-07-26 23:19:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000957702s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.............X.........X...........X.......XXXXXXXXXXXXXXXXX] duration: 30.231469424s FAIL (2025-07-26 23:18:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FYZX2IJKW7V4LF27E57JI5Q, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637222606 (59h42m36.359785646s) [noise=true , address=104.36.82.10:4078 , piece_id=GOG2ELJV7PASKKLIQL66SZNPDVNI7CS7PZYIKMYJFUE7WL2BHQ3Q, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637582695 (59h42m36.360145735s) [noise=true , address=77.90.31.14:4663 , piece_id=XOI5ZJNMDK7ECJOITJ4FKGYRSUE4K4TVGTHYREUTMQX6IGQBU27Q, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637598786 (59h42m36.360161817s) [noise=true , address=104.36.82.4:4804 , piece_id=SJ2XYJ33TRYMZ76YPEBTHR5BCGXINKFO7TZWYQH2XWNVBMD6YF6A, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637852381 (59h42m36.360415422s) [noise=true , address=104.36.82.6:4874 , piece_id=WNPITA6QHRV36U6FZJCXNE6ZSXHX3MPMPXZOIXQ3RMUR426FOCVQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637907337 (59h42m36.360470377s) [noise=true , address=104.36.82.5:4838 , piece_id=E253BSYUKXW66GO3VQCYLRE3OMOD5FPS7JMXJ4FF4HPT3KXC5MBQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638112459 (59h42m36.360675489s) [noise=true , address=104.36.82.7:4909 , piece_id=RVI2I5VLI7KN7AO3JO7X6O75XY3LVJU22NIIT52B4Q63QUDPB3TA, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638202291 (59h42m36.360765331s) [noise=true , address=104.36.82.9:4005 , piece_id=Y6YHHK5WVJZPSAH5RK4O55F2FD75VCGLJBK7MRUMGEZQRDRUFLBQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638362627 (59h42m36.360925667s) [noise=true , address=77.90.31.10:4518 , piece_id=UKCAYXKYD6CFUEANJNEA2WVW7GC4E7UYZSZGXYF6QBLMEL54MRKQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638511312 (59h42m36.361074352s) [noise=true , address=104.36.82.14:4223 , piece_id=YYLEHMNMM4VXPL2QS5VBHNDCMQZUG5G3UQF4F6LFBFRIOWYY67NQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638636401 (59h42m36.361199431s) [noise=true , address=45.91.116.7:4009 , piece_id=4GXRRUQYDEZKH55WM75GPGGW72B3BM52JFI23SW3WFFCEMKLGNHQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638814312 (59h42m36.361377362s) [noise=true , address=104.36.82.4:4803 , piece_id=VG5NDUJOFY36VQCWSPKMQ2EGWA73JLQVYS2HHV3IWZ5VXBKP3LGA, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638904093 (59h42m36.361467123s) [noise=true , address=45.91.116.13:4154 , piece_id=ENPUWANJ4IFFG5UMDJFN7IAQBLQE7IDS6FD5WDIHH575KZHCLRDA, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.639138212 (59h42m36.361701252s) [noise=true , address=45.91.116.5:4298 , piece_id=MUH4L5JURP6CRJYMZV5AXF3ZDX23SD4XMGYXLFTLSKZS536H7POQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.639342332 (59h42m36.361905362s) [noise=true , address=104.36.82.8:4969 , piece_id=6E4DJXDCDCLKE7STRF65PE7GCWYG6HCNCRNBUGWO7VF2MROTT7OQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.639815247 (59h42m36.362378287s) [noise=true , address=104.36.82.15:4258 , piece_id=SMMNKQOQIDYBTEL5M2XXGQVNWVK3JMA3XI3E6DIFDD4TAM5EQSCA, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:28.329056920 (59h42m38.05161996s) [upload=57286]: Next returned nil segment Jul 26 23:18:28.329072469 (59h42m38.051635509s) [upload=57286]: Waiting for error group managing segments... Jul 26 23:18:28.659986910 (59h42m38.38254995s) [noise=true , address=45.91.116.5:4298 , piece_id=MUH4L5JURP6CRJYMZV5AXF3ZDX23SD4XMGYXLFTLSKZS536H7POQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: <nil> Jul 26 23:18:28.662276450 (59h42m38.38483948s) [noise=true , address=45.91.116.13:4154 , piece_id=ENPUWANJ4IFFG5UMDJFN7IAQBLQE7IDS6FD5WDIHH575KZHCLRDA, seg_pos={0 0}, upload=57286]: Done uploading piece. err: <nil> Jul 26 23:18:28.740573276 (59h42m38.463136316s) [noise=true , address=77.90.31.14:4663 , piece_id=XOI5ZJNMDK7ECJOITJ4FKGYRSUE4K4TVGTHYREUTMQX6IGQBU27Q, seg_pos={0 0}, upload=57286]: Done uploading piece. err: <nil> Jul 26 23:18:28.846458196 (59h42m38.569021236s) [noise=true , address=45.91.116.7:4009 , piece_id=4GXRRUQYDEZKH55WM75GPGGW72B3BM52JFI23SW3WFFCEMKLGNHQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: <nil> Jul 26 23:18:55.758542517 (59h43m5.481105557s) [noise=true , address=104.36.82.15:4258 , piece_id=SMMNKQOQIDYBTEL5M2XXGQVNWVK3JMA3XI3E6DIFDD4TAM5EQSCA, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758587283 (59h43m5.481150313s) [noise=true , address=104.36.82.6:4874 , piece_id=WNPITA6QHRV36U6FZJCXNE6ZSXHX3MPMPXZOIXQ3RMUR426FOCVQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758618432 (59h43m5.481181462s) [noise=true , address=104.36.82.14:4223 , piece_id=YYLEHMNMM4VXPL2QS5VBHNDCMQZUG5G3UQF4F6LFBFRIOWYY67NQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758647979 (59h43m5.481211019s) [noise=true , address=104.36.82.9:4005 , piece_id=Y6YHHK5WVJZPSAH5RK4O55F2FD75VCGLJBK7MRUMGEZQRDRUFLBQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758699417 (59h43m5.481262447s) [noise=true , address=104.36.82.4:4803 , piece_id=VG5NDUJOFY36VQCWSPKMQ2EGWA73JLQVYS2HHV3IWZ5VXBKP3LGA, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758737940 (59h43m5.48130098s) [noise=true , address=104.36.82.8:4969 , piece_id=6E4DJXDCDCLKE7STRF65PE7GCWYG6HCNCRNBUGWO7VF2MROTT7OQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758788257 (59h43m5.481351297s) [noise=true , address=104.36.82.10:4078 , piece_id=GOG2ELJV7PASKKLIQL66SZNPDVNI7CS7PZYIKMYJFUE7WL2BHQ3Q, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758858852 (59h43m5.481421892s) [noise=true , address=104.36.82.13:4185 , piece_id=4ILYMZHGBXFKP3AHC6XEH62VMMVUTFYZX2IJKW7V4LF27E57JI5Q, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:46998->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758894130 (59h43m5.48145717s) [noise=true , address=104.36.82.5:4838 , piece_id=E253BSYUKXW66GO3VQCYLRE3OMOD5FPS7JMXJ4FF4HPT3KXC5MBQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:50440->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758913687 (59h43m5.481476727s) [noise=true , address=104.36.82.7:4909 , piece_id=RVI2I5VLI7KN7AO3JO7X6O75XY3LVJU22NIIT52B4Q63QUDPB3TA, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758926221 (59h43m5.481489261s) [noise=true , address=104.36.82.4:4804 , piece_id=SJ2XYJ33TRYMZ76YPEBTHR5BCGXINKFO7TZWYQH2XWNVBMD6YF6A, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758964525 (59h43m5.481527565s) [noise=true , address=77.90.31.10:4518 , piece_id=UKCAYXKYD6CFUEANJNEA2WVW7GC4E7UYZSZGXYF6QBLMEL54MRKQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 188.166.225.170:39552->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758977619 (59h43m5.481540659s) [seg_pos={0 0}, upload=57286]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Jul 26 23:18:55.988191826 (59h43m5.710754856s) [upload=57286]: Done uploading segments... Jul 26 23:18:55.988244526 (59h43m5.710807567s) [upload=57286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.231481339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230501828s FAIL (2025-07-26 23:18:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== address=104.36.82.55:4215 , piece_id=HROJI66WXDCYQBNP7GNWZEVWKNY2XDQHHNGKMERNGTCWIZ7C4S5Q, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.582172539 (59h41m49.304735579s) [noise=true , address=77.90.31.16:4737 , piece_id=53SFWGTX7ZQHZFG5PG2AOCYABZ7CAMSYNHR36UAMWIGY244KUPVA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.582236822 (59h41m49.304799872s) [noise=true , address=79.127.205.233:4655 , piece_id=P3JGPQ3J3RDO2745RRM5BOCVHDO2W7ET66H3ADGHB625VL3ETKKA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.582991426 (59h41m49.305554516s) [noise=true , address=77.90.31.13:4636 , piece_id=MBGEVNEPGQ474FQHFIICLAFWBHRIE2F3IFMZMJTZNA3YC3RJ7KOQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.583472066 (59h41m49.306035116s) [noise=true , address=104.36.82.47:4543 , piece_id=25L4ITRFE3WLY5STEWUYEAGSVCREJOHHGOGMG5Q2GEPAIC6DEY4A, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.583942856 (59h41m49.306505896s) [noise=true , address=104.36.82.19:4402 , piece_id=MNDMEK6QYHSPUO3CTDEO4MQVIYHZU34G5IGYODK6RF5TGGAGCLOQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.584403968 (59h41m49.306967018s) [noise=true , address=104.36.82.46:4513 , piece_id=5RVJHBCSVJRRVFWDELFEBP746NCGFSVNSACCZNJX3SJGXAT2TNWA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.584782613 (59h41m49.307345653s) [noise=true , address=77.90.31.11:4579 , piece_id=FPIZFOWNQWOT4WB5C4PHQEZWJID4HT2M4EQFTXZR25GEDRSRPERA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.585152340 (59h41m49.307715381s) [noise=true , address=104.36.82.52:4099 , piece_id=FHW74NJHGJO3PDCVOJFJ3FHXADVLTN4OKBCGPRPFACSLRGGRUILQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.585440752 (59h41m49.308003792s) [noise=true , address=104.36.82.53:4141 , piece_id=VIW33FWGS52QA5PNTCVA7EG4G3NYHOZP4O3PNZIVMAYH6F5AW72Q, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.585752478 (59h41m49.308315518s) [noise=true , address=79.127.205.245:4249 , piece_id=UCWUBRUFRK5JS3IIHI6YYRFORSIHZSDODZC3DAFPDW4W6EU36OAA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.586148456 (59h41m49.308711496s) [noise=true , address=79.127.205.234:4615 , piece_id=WFCANOT4STSC3F4BHSOUROVL6QSLVSXEH2VQIF32GCFXZUAOGHNQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.586427981 (59h41m49.308991021s) [noise=true , address=64.49.13.140:4514 , piece_id=QRJJYRYMNQSIVIRGCYU7WC3T7RKGTPQL5VKY4UASK6YAZYMLDNCQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.586699861 (59h41m49.309262901s) [noise=true , address=64.49.13.138:4482 , piece_id=RLB35JTBM3UTZWWPZC7WLK53LDR3TOFYSVGPGEK3UYRR2KHDMBLA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.587004033 (59h41m49.309567073s) [noise=true , address=104.36.82.54:4185 , piece_id=JVS64522A35LTOMIBRP4VPOE2F24KBYC2LAXKVX3BVK54NNDKW7A, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.587296303 (59h41m49.309859343s) [noise=true , address=77.90.31.18:4828 , piece_id=5CT6S2LHXUECI3CVKFLJGJLDQYHQLJOBK4WCMIDUSEZ6KW3CKRSQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.587557341 (59h41m49.310120382s) [noise=true , address=79.127.219.38:4004 , piece_id=TTZFRFMYKQUNE3JTLEKG75D47GV3AKR4ASIRX56SXMZXFKTHXDYQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.588280105 (59h41m49.310843155s) [noise=true , address=104.36.82.56:4259 , piece_id=LZAQU5TFCKAP6GO2MMRB3BHB7VDSFMLBO5WMQF3XTWIPKPV5H7VA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:40.697095606 (59h41m50.419658646s) [upload=57272]: Next returned nil segment Jul 26 23:17:40.697110614 (59h41m50.419673654s) [upload=57272]: Waiting for error group managing segments... Jul 26 23:17:40.850969162 (59h41m50.573532202s) [noise=true , address=64.49.13.138:4482 , piece_id=RLB35JTBM3UTZWWPZC7WLK53LDR3TOFYSVGPGEK3UYRR2KHDMBLA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:17:40.983070762 (59h41m50.705633802s) [noise=true , address=79.127.205.234:4615 , piece_id=WFCANOT4STSC3F4BHSOUROVL6QSLVSXEH2VQIF32GCFXZUAOGHNQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.035315874 (59h41m50.757878914s) [noise=true , address=79.127.205.245:4249 , piece_id=UCWUBRUFRK5JS3IIHI6YYRFORSIHZSDODZC3DAFPDW4W6EU36OAA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.248241625 (59h41m50.970804665s) [noise=true , address=79.127.205.233:4655 , piece_id=P3JGPQ3J3RDO2745RRM5BOCVHDO2W7ET66H3ADGHB625VL3ETKKA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.466347097 (59h41m51.188910137s) [noise=true , address=77.90.31.18:4828 , piece_id=5CT6S2LHXUECI3CVKFLJGJLDQYHQLJOBK4WCMIDUSEZ6KW3CKRSQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.495291166 (59h41m51.217854206s) [noise=true , address=79.127.219.38:4004 , piece_id=TTZFRFMYKQUNE3JTLEKG75D47GV3AKR4ASIRX56SXMZXFKTHXDYQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.537493632 (59h41m51.260056672s) [noise=true , address=77.90.31.16:4737 , piece_id=53SFWGTX7ZQHZFG5PG2AOCYABZ7CAMSYNHR36UAMWIGY244KUPVA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.545015775 (59h41m51.267578815s) [noise=true , address=104.36.82.54:4185 , piece_id=JVS64522A35LTOMIBRP4VPOE2F24KBYC2LAXKVX3BVK54NNDKW7A, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.594943411 (59h41m51.317506441s) [noise=true , address=77.90.31.11:4579 , piece_id=FPIZFOWNQWOT4WB5C4PHQEZWJID4HT2M4EQFTXZR25GEDRSRPERA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:42.524604555 (59h41m52.247167595s) [noise=true , address=64.49.13.140:4514 , piece_id=QRJJYRYMNQSIVIRGCYU7WC3T7RKGTPQL5VKY4UASK6YAZYMLDNCQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:18:02.277063189 (59h42m11.99962626s) [noise=true , address=77.90.31.13:4636 , piece_id=MBGEVNEPGQ474FQHFIICLAFWBHRIE2F3IFMZMJTZNA3YC3RJ7KOQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:18:38.424884073 (59h42m48.147447113s) [noise=true , address=104.36.82.46:4513 , piece_id=5RVJHBCSVJRRVFWDELFEBP746NCGFSVNSACCZNJX3SJGXAT2TNWA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: rpc: dial tcp 104.36.82.46:4513: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.424929109 (59h42m48.147492149s) [noise=true , address=104.36.82.47:4543 , piece_id=25L4ITRFE3WLY5STEWUYEAGSVCREJOHHGOGMG5Q2GEPAIC6DEY4A, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: rpc: dial tcp 104.36.82.47:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.424942846 (59h42m48.147505886s) [noise=true , address=104.36.82.55:4215 , piece_id=HROJI66WXDCYQBNP7GNWZEVWKNY2XDQHHNGKMERNGTCWIZ7C4S5Q, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 188.166.225.170:41890->104.36.82.55:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.424966892 (59h42m48.147529932s) [noise=true , address=104.36.82.56:4259 , piece_id=LZAQU5TFCKAP6GO2MMRB3BHB7VDSFMLBO5WMQF3XTWIPKPV5H7VA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: write tcp 188.166.225.170:49700->104.36.82.56:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.424996037 (59h42m48.147559078s) [noise=true , address=104.36.82.19:4402 , piece_id=MNDMEK6QYHSPUO3CTDEO4MQVIYHZU34G5IGYODK6RF5TGGAGCLOQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:104.36.82.19:4402): protocol: rpc: dial tcp 104.36.82.19:4402: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.425032477 (59h42m48.147595517s) [noise=true , address=104.36.82.52:4099 , piece_id=FHW74NJHGJO3PDCVOJFJ3FHXADVLTN4OKBCGPRPFACSLRGGRUILQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT, address:104.36.82.52:4099): protocol: rpc: dial tcp 104.36.82.52:4099: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.425061733 (59h42m48.147624773s) [noise=true , address=104.36.82.53:4141 , piece_id=VIW33FWGS52QA5PNTCVA7EG4G3NYHOZP4O3PNZIVMAYH6F5AW72Q, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: write tcp 188.166.225.170:55394->104.36.82.53:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.425080128 (59h42m48.147643168s) [seg_pos={0 0}, upload=57272]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:18:38.653644601 (59h42m48.376207641s) [upload=57272]: Done uploading segments... Jul 26 23:18:38.653685079 (59h42m48.376248109s) [upload=57272]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230517077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [............X..XX...X.X.X.X.......XX..XX.XX..XXX............] duration: 19.430575873s FAIL (2025-07-26 23:18:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.43059578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..............X.........X...........X.......XXXXXXXXXXXXXXXX] duration: 30.236722725s FAIL (2025-07-26 23:18:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== east 13 but got 9) Jul 26 23:17:55.945461463 (59h42m5.668024493s) [upload=57278]: Starting upload Jul 26 23:17:55.945518392 (59h42m5.668081432s) [upload=57278]: Done starting upload Jul 26 23:17:55.945571363 (59h42m5.668134403s) [upload=57278]: Uploading segments... Jul 26 23:17:55.945689339 (59h42m5.668252379s) [upload=57278]: Got next segment. Inline: false Jul 26 23:17:55.945715268 (59h42m5.668278308s) [upload=57278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:17:55.945719506 (59h42m5.668282546s) [upload=57278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:17:56.180132761 (59h42m5.902695801s) [seg_pos={0 0}, upload=57278]: Begin upload segment... Jul 26 23:17:56.180171965 (59h42m5.902735005s) [seg_pos={0 0}, upload=57278]: Done begin upload segment. Jul 26 23:17:56.180239425 (59h42m5.902802465s) [noise=true , address=104.36.82.3:4767 , piece_id=4VYMQPS34INP5Q2PYW6FO2DGP4Z6ZSF6BIERUU26DQFDE2X5ZBCQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.180285372 (59h42m5.902848412s) [noise=true , address=45.91.116.6:4975 , piece_id=FMBIGDRVIX4363YSFCGHU5B74LWT7Y63E3UAHBPGQG3RTU4YAVBQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.180729111 (59h42m5.903292151s) [noise=true , address=104.36.82.10:4079 , piece_id=2JGXXC7DW3PFF3WZU7CU7S4MXAJ4ICMHSZJ4HKY33XC7EEPLLQEA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.180755902 (59h42m5.903318942s) [noise=true , address=45.91.116.7:4010 , piece_id=P5BQ4ECAVUJO445S2XQ4AISX6VYFIJ42QHKICROMPFEZNQU7J4GA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181014929 (59h42m5.903577979s) [noise=true , address=104.36.82.11:4115 , piece_id=T75TOGWKL3PYYSBQBMTYJZCOKJBIIGHYUZFEUTWXJWAMSY4PYSFA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181087889 (59h42m5.903650929s) [noise=true , address=104.36.82.13:4185 , piece_id=RJZXED5DVMIC7RQMBZX6THKHNYPQUXHHOD4GNOTYEETGGQ2RTQMA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181283713 (59h42m5.903846753s) [noise=true , address=45.91.116.10:4082 , piece_id=XFKFSCMKDL2S5TDTXH5MUL2UCP3BG56ISKIQDAHK64ZAWL2XNWBQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181370559 (59h42m5.903933599s) [noise=true , address=104.36.82.7:4910 , piece_id=M2VYAEIAYPWTKC6UHNFFPR4SOSPBUUUN7KWYXBMHLZURD4EVVH3Q, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181529814 (59h42m5.904092844s) [noise=true , address=104.36.82.15:4257 , piece_id=UA3HB6N2HXPKQIBYZO7B2HMTP5X426VHT65MPAKHHDU7P3IVZJRA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181667297 (59h42m5.904230337s) [noise=true , address=45.91.116.4:4937 , piece_id=5NW5P4JPWXD3OIWOOOUF63JMRNUVZZMLS2TE4IBJEU75CFJMYKAA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181786865 (59h42m5.904349906s) [noise=true , address=77.90.31.17:4770 , piece_id=HZB2R73FJCWPNQSARFBARD2MZXIO2VRAYGU4OHJ7IMJRURJ2UGIA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.182067332 (59h42m5.904630362s) [noise=true , address=45.91.116.14:4191 , piece_id=XJCD6V2UBXDCJCQVHGCOVLWG6YTAK3RDUM33PJILYH2E3KJT4DEA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.182261634 (59h42m5.904824674s) [noise=true , address=104.36.82.13:4187 , piece_id=RKCOQPWQKJAIA6V3MWOS4WGGFRZ7YAXGCPWGJBZ7AKDW6ECMC7OQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.182455705 (59h42m5.905018745s) [noise=true , address=45.91.116.6:4973 , piece_id=PKO7CGBRCKQEEA2QB5SPSDKEG55X3JXWLYWJQCZSWRJ5H6CSNBDQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.183891642 (59h42m5.906454682s) [noise=true , address=104.36.82.2:4730 , piece_id=MUMTIPJSM2X2Q73IGPRXNU774EXZ7LZQ4AMB3IHRA2OT4NBCEJHA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.184137723 (59h42m5.906700753s) [noise=true , address=104.36.82.10:4077 , piece_id=4UF2CDXZFI7LF2QR6AGQUCFDOWZLCU26J4RWRIUSQ4BLHEFUPV3A, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:57.984487703 (59h42m7.707050743s) [upload=57278]: Next returned nil segment Jul 26 23:17:57.984502241 (59h42m7.707065271s) [upload=57278]: Waiting for error group managing segments... Jul 26 23:17:58.280634973 (59h42m8.003198003s) [noise=true , address=45.91.116.6:4973 , piece_id=PKO7CGBRCKQEEA2QB5SPSDKEG55X3JXWLYWJQCZSWRJ5H6CSNBDQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.293320285 (59h42m8.015883325s) [noise=true , address=45.91.116.10:4082 , piece_id=XFKFSCMKDL2S5TDTXH5MUL2UCP3BG56ISKIQDAHK64ZAWL2XNWBQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.300193614 (59h42m8.022756654s) [noise=true , address=45.91.116.7:4010 , piece_id=P5BQ4ECAVUJO445S2XQ4AISX6VYFIJ42QHKICROMPFEZNQU7J4GA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.300569674 (59h42m8.023132704s) [noise=true , address=45.91.116.6:4975 , piece_id=FMBIGDRVIX4363YSFCGHU5B74LWT7Y63E3UAHBPGQG3RTU4YAVBQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.364854361 (59h42m8.087417401s) [noise=true , address=45.91.116.4:4937 , piece_id=5NW5P4JPWXD3OIWOOOUF63JMRNUVZZMLS2TE4IBJEU75CFJMYKAA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.429865256 (59h42m8.152428296s) [noise=true , address=45.91.116.14:4191 , piece_id=XJCD6V2UBXDCJCQVHGCOVLWG6YTAK3RDUM33PJILYH2E3KJT4DEA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:18:14.206751585 (59h42m23.929314675s) [noise=true , address=104.36.82.13:4187 , piece_id=RKCOQPWQKJAIA6V3MWOS4WGGFRZ7YAXGCPWGJBZ7AKDW6ECMC7OQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:18:14.321264263 (59h42m24.043827303s) [noise=true , address=104.36.82.13:4185 , piece_id=RJZXED5DVMIC7RQMBZX6THKHNYPQUXHHOD4GNOTYEETGGQ2RTQMA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:18:25.296932302 (59h42m35.019495342s) [noise=true , address=104.36.82.7:4910 , piece_id=M2VYAEIAYPWTKC6UHNFFPR4SOSPBUUUN7KWYXBMHLZURD4EVVH3Q, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.296977869 (59h42m35.019540909s) [noise=true , address=104.36.82.11:4115 , piece_id=T75TOGWKL3PYYSBQBMTYJZCOKJBIIGHYUZFEUTWXJWAMSY4PYSFA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297010061 (59h42m35.019573101s) [noise=true , address=104.36.82.15:4257 , piece_id=UA3HB6N2HXPKQIBYZO7B2HMTP5X426VHT65MPAKHHDU7P3IVZJRA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297046590 (59h42m35.01960963s) [noise=true , address=104.36.82.2:4730 , piece_id=MUMTIPJSM2X2Q73IGPRXNU774EXZ7LZQ4AMB3IHRA2OT4NBCEJHA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:58774->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297209792 (59h42m35.019772842s) [noise=true , address=77.90.31.17:4770 , piece_id=HZB2R73FJCWPNQSARFBARD2MZXIO2VRAYGU4OHJ7IMJRURJ2UGIA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297253296 (59h42m35.019816336s) [noise=true , address=104.36.82.10:4077 , piece_id=4UF2CDXZFI7LF2QR6AGQUCFDOWZLCU26J4RWRIUSQ4BLHEFUPV3A, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297290978 (59h42m35.019854018s) [noise=true , address=104.36.82.3:4767 , piece_id=4VYMQPS34INP5Q2PYW6FO2DGP4Z6ZSF6BIERUU26DQFDE2X5ZBCQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 188.166.225.170:55208->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297316256 (59h42m35.019879296s) [noise=true , address=104.36.82.10:4079 , piece_id=2JGXXC7DW3PFF3WZU7CU7S4MXAJ4ICMHSZJ4HKY33XC7EEPLLQEA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297374718 (59h42m35.019937768s) [seg_pos={0 0}, upload=57278]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:18:25.531540137 (59h42m35.254103177s) [upload=57278]: Done uploading segments... Jul 26 23:18:25.531566388 (59h42m35.254129428s) [upload=57278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.236744456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X.X........X.X...........................XXXXXXXXXXXXXX] duration: 30.234423615s FAIL (2025-07-26 23:18:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.23444704s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.835386774s FAIL (2025-07-26 23:18:17 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.835411923s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XXX.X..X.....X] duration: 30.000760786s FAIL (2025-07-26 23:18:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000776927s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X.X........X.X...........................XXXXXXXXXXXXX] duration: 30.239286738s FAIL (2025-07-26 23:17:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.239307787s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.85852748s FAIL (2025-07-26 23:17:43 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.858546685s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235189012s FAIL (2025-07-26 23:17:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== iece_id=P2EB4FTEHYZNQOTEZUYJCKWPGGYI4OKTLLTYNLY6VOKSNBQDZMOA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.101246508 (59h40m48.823809548s) [noise=true , address=79.127.201.215:4360 , piece_id=HA5SDSTZT66XA3RYMDB7Y5NE44Y3P65LOWCFPW2O7D4BO5J26ZPA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.101605725 (59h40m48.824168765s) [noise=true , address=79.127.201.214:4321 , piece_id=OIL42A7QVMSCPEZI6D6YSPTTUEAFEHZUPZU7TELG5KYWYY3ADY3A, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.101652835 (59h40m48.824215875s) [noise=true , address=104.36.82.55:4194 , piece_id=CIYWLFM242PBGYBNJTFDSQJA4TVDENRNFZCTAYLBDNKM2MO2MR4Q, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.102153032 (59h40m48.824716072s) [noise=true , address=104.36.82.54:4191 , piece_id=LBQNYW34JFXHISZWAWOE55ISPDBVW3PJAXUT5VUNYEJPALCUNKTA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.102164485 (59h40m48.824727515s) [noise=true , address=79.127.205.235:4682 , piece_id=XEWSYPC4CUBC4GNDWXFHMNNOZGAC7K7NRJFFQV4LR7PK55XFDKMQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.102439030 (59h40m48.82500207s) [noise=true , address=79.127.219.38:4017 , piece_id=2L27YX7JSSW5U2ZKEOFMKLUR2SJWOYN6OUNFZIVQVJPPZGRRJH7Q, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.103321278 (59h40m48.825884318s) [noise=true , address=104.36.82.56:4253 , piece_id=CZEF6CF23HOROPA3AORZP46TKK4H7GTMEXSEX6ONHCNAPYNIFQGA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.103457328 (59h40m48.826020358s) [noise=true , address=79.127.205.245:4240 , piece_id=AMHSGVZC3NYB6SW6F3WQTCR4GLT76EFUNRPF53N7N4I3FILILT3Q, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.103958337 (59h40m48.826521367s) [noise=true , address=104.36.82.52:4091 , piece_id=RBWO4I6MEO2ZNHDFP6WY4GZ6BVP2CTGYNEC6RDJE33OOE2WKTYYQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.104324697 (59h40m48.826887737s) [noise=true , address=77.90.31.15:4723 , piece_id=SEFPWKWXVERRJ2TINBTDG7GMYK3YY7WGHNGZKW2WECRHNLY52CFQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.104343744 (59h40m48.826906784s) [noise=true , address=104.36.82.47:4530 , piece_id=ABBT2EMARL5LDMMBQKMGHKAPSFQD26T4FV5JEOHZF5RPJZV2KR5A, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.104601607 (59h40m48.827164647s) [noise=true , address=77.90.31.16:4743 , piece_id=4DWFOOWEY3QESIYUILZNTYVS6JY4AM3NZTJGJLXDIKB5L3EUEPHA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.104849772 (59h40m48.827412812s) [noise=true , address=104.36.82.46:4514 , piece_id=LUC4XUVBSPO66OT7M3VP3D3EBRIOOUGBI6MLJAFQ5ESAPAFD2QEQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.105089110 (59h40m48.82765215s) [noise=true , address=45.91.116.9:4273 , piece_id=NKAWJ5FDUUQWLRHZUNDTQF5O7F5GDGI6FVU2TRKWKATTIGE5246Q, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.105313409 (59h40m48.827876449s) [noise=true , address=104.36.82.41:4774 , piece_id=EO5GFI4ZPCYR5V3GZLUFNYCGRO2SYJXKE6PWHL4MPZS7J6FO2YTQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.105531607 (59h40m48.828094647s) [noise=true , address=45.91.116.11:4124 , piece_id=7LOHMHKC7JLNVDGXEKB5CYXR5ZOPT3Q5OZ2LT4RARSJPNTN267BA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.105759943 (59h40m48.828322983s) [noise=true , address=45.91.116.9:4270 , piece_id=B7YU2K7RSU2PLMK54AKIFJU5YA5ZHBXNDVX3PTTKKK5WK3ZLIAWA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:40.110039756 (59h40m49.832602796s) [noise=true , address=79.127.201.214:4321 , piece_id=OIL42A7QVMSCPEZI6D6YSPTTUEAFEHZUPZU7TELG5KYWYY3ADY3A, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:16:40.220391383 (59h40m49.942954423s) [upload=57256]: Next returned nil segment Jul 26 23:16:40.220409257 (59h40m49.942972297s) [upload=57256]: Waiting for error group managing segments... Jul 26 23:16:40.505659167 (59h40m50.228222207s) [noise=true , address=79.127.205.235:4682 , piece_id=XEWSYPC4CUBC4GNDWXFHMNNOZGAC7K7NRJFFQV4LR7PK55XFDKMQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:40.574667831 (59h40m50.297230871s) [noise=true , address=79.127.205.245:4240 , piece_id=AMHSGVZC3NYB6SW6F3WQTCR4GLT76EFUNRPF53N7N4I3FILILT3Q, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:40.967035320 (59h40m50.68959836s) [noise=true , address=79.127.219.38:4017 , piece_id=2L27YX7JSSW5U2ZKEOFMKLUR2SJWOYN6OUNFZIVQVJPPZGRRJH7Q, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:41.094575093 (59h40m50.817138133s) [noise=true , address=77.90.31.15:4723 , piece_id=SEFPWKWXVERRJ2TINBTDG7GMYK3YY7WGHNGZKW2WECRHNLY52CFQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:41.153069600 (59h40m50.87563263s) [noise=true , address=45.91.116.9:4270 , piece_id=B7YU2K7RSU2PLMK54AKIFJU5YA5ZHBXNDVX3PTTKKK5WK3ZLIAWA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:41.269502842 (59h40m50.992065882s) [noise=true , address=77.90.31.16:4743 , piece_id=4DWFOOWEY3QESIYUILZNTYVS6JY4AM3NZTJGJLXDIKB5L3EUEPHA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:41.279714380 (59h40m51.00227742s) [noise=true , address=45.91.116.9:4273 , piece_id=NKAWJ5FDUUQWLRHZUNDTQF5O7F5GDGI6FVU2TRKWKATTIGE5246Q, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:45.000563296 (59h40m54.723126336s) [noise=true , address=45.91.116.11:4124 , piece_id=7LOHMHKC7JLNVDGXEKB5CYXR5ZOPT3Q5OZ2LT4RARSJPNTN267BA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:45.732897706 (59h40m55.455460746s) [noise=true , address=79.127.220.101:4048 , piece_id=P2EB4FTEHYZNQOTEZUYJCKWPGGYI4OKTLLTYNLY6VOKSNBQDZMOA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:17:05.714163611 (59h41m15.436726661s) [noise=true , address=79.127.201.215:4360 , piece_id=HA5SDSTZT66XA3RYMDB7Y5NE44Y3P65LOWCFPW2O7D4BO5J26ZPA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:17:37.968191773 (59h41m47.690754813s) [noise=true , address=104.36.82.46:4514 , piece_id=LUC4XUVBSPO66OT7M3VP3D3EBRIOOUGBI6MLJAFQ5ESAPAFD2QEQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: rpc: dial tcp 104.36.82.46:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968261927 (59h41m47.690824967s) [noise=true , address=104.36.82.56:4253 , piece_id=CZEF6CF23HOROPA3AORZP46TKK4H7GTMEXSEX6ONHCNAPYNIFQGA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: write tcp 188.166.225.170:59058->104.36.82.56:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968307314 (59h41m47.690870354s) [noise=true , address=104.36.82.55:4194 , piece_id=CIYWLFM242PBGYBNJTFDSQJA4TVDENRNFZCTAYLBDNKM2MO2MR4Q, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: rpc: dial tcp 104.36.82.55:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968346399 (59h41m47.690909429s) [noise=true , address=104.36.82.47:4530 , piece_id=ABBT2EMARL5LDMMBQKMGHKAPSFQD26T4FV5JEOHZF5RPJZV2KR5A, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: write tcp 188.166.225.170:35900->104.36.82.47:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968369183 (59h41m47.690932223s) [noise=true , address=104.36.82.54:4191 , piece_id=LBQNYW34JFXHISZWAWOE55ISPDBVW3PJAXUT5VUNYEJPALCUNKTA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: write tcp 188.166.225.170:42504->104.36.82.54:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968381686 (59h41m47.690944716s) [noise=true , address=104.36.82.52:4091 , piece_id=RBWO4I6MEO2ZNHDFP6WY4GZ6BVP2CTGYNEC6RDJE33OOE2WKTYYQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: write tcp 188.166.225.170:33386->104.36.82.52:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968392928 (59h41m47.690955968s) [noise=true , address=104.36.82.41:4774 , piece_id=EO5GFI4ZPCYR5V3GZLUFNYCGRO2SYJXKE6PWHL4MPZS7J6FO2YTQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb, address:104.36.82.41:4774): protocol: rpc: dial tcp 104.36.82.41:4774: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968409629 (59h41m47.69097267s) [seg_pos={0 0}, upload=57256]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:17:38.201427694 (59h41m47.923990734s) [upload=57256]: Done uploading segments... Jul 26 23:17:38.201453774 (59h41m47.924016804s) [upload=57256]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.235200333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.............X..XX...X.X.X.X.......XX..XX.XX..XXX...........] duration: 10.119631004s FAIL (2025-07-26 23:17:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.119641414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXX.X..X.....] duration: 26.164022328s FAIL (2025-07-26 23:17:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.164040662s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [..............X..XX...X.X.X.X.......XX..XX.XX..XXX..........] duration: 30.898734644s FAIL (2025-07-26 23:17:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.898751226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...............X.........X...........X.......XXXXXXXXXXXXXXX] duration: 30.240002439s FAIL (2025-07-26 23:17:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 47.996956281 (59h40m57.719519311s) [upload=57250]: Done uploading segments... Jul 26 23:16:47.996981249 (59h40m57.719544289s) [upload=57250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:16:48.872851333 (59h40m58.595414373s) [upload=57260]: Starting upload Jul 26 23:16:48.872927048 (59h40m58.595490088s) [upload=57260]: Done starting upload Jul 26 23:16:48.873029574 (59h40m58.595592614s) [upload=57260]: Uploading segments... Jul 26 23:16:48.873142911 (59h40m58.595705951s) [upload=57260]: Got next segment. Inline: false Jul 26 23:16:48.873180984 (59h40m58.595744024s) [upload=57260]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:16:48.873187096 (59h40m58.595750126s) [upload=57260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:16:49.343403844 (59h40m59.065966884s) [seg_pos={0 0}, upload=57260]: Begin upload segment... Jul 26 23:16:49.343447838 (59h40m59.066010878s) [seg_pos={0 0}, upload=57260]: Done begin upload segment. Jul 26 23:16:49.343470662 (59h40m59.066033692s) [noise=true , address=45.91.116.7:4011 , piece_id=MXU2WJATFMMHQUHMFCHPHPMI7QR2R5D67PHLO2M5KFN3V6VHC72A, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.343514806 (59h40m59.066077846s) [noise=true , address=77.90.31.14:4661 , piece_id=4ZOGBM2UN4LVQMI26S3VSKNFQNRJE5UGL3AZEIRFPZZ7QL63QYIA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.343834508 (59h40m59.066397538s) [noise=true , address=45.91.116.8:4045 , piece_id=KJ3EZGGLW3DDDXUVWXAQIF5NIWUC2EVAPUG33TAW4D2YN75OKPOA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.344250774 (59h40m59.066813814s) [noise=true , address=104.36.82.10:4078 , piece_id=4L4Y4KPKVXESEYSZP6Q26R5J7V2CSXUNEI2SE7GB7B3A3OJBC4VQ, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.344703571 (59h40m59.067266611s) [noise=true , address=45.91.116.12:4226 , piece_id=LYRTQQKREHKRKBVPFOR3AVIXQB6SVJT4ZECKKOH23WNRMZTGO4OA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.344823220 (59h40m59.0673863s) [noise=true , address=104.36.82.10:4079 , piece_id=7IMEBZTMVUAAX2MY6HU3FSZJSDT72RHJY4FYP2RWVWHFRBDVJZAA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.344968708 (59h40m59.067531748s) [noise=true , address=45.91.116.10:4083 , piece_id=EK2S656KNAC7MSJGYK3FAL4ZJOBP4FF26VRPXHKPJGWJLM7JQFAQ, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345214779 (59h40m59.067777819s) [noise=true , address=77.90.31.10:4518 , piece_id=SJSEZPJQNAV4NXEMIOCNKZTW6RR7HXHWZ7C3BWJAEOJA3OSFPHTQ, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345458385 (59h40m59.068021425s) [noise=true , address=104.36.82.6:4876 , piece_id=IQB4KI5VC3UXWREOJLYV3FOEEPEZ42YTQI7W73Z37ZTQYCG2MK4A, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345689457 (59h40m59.068252497s) [noise=true , address=45.91.116.7:4009 , piece_id=AWFZDBRE4HGD3BOPUU7JITDWE4Y67W3CV7U4SH5BDFNQILUZA6SA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345922343 (59h40m59.068485383s) [noise=true , address=45.91.116.5:4298 , piece_id=X6DEY45NFWRHQPHUODFEB7WFWRKE6VIACDRJE2D2L4VYXHIWQEVA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345946228 (59h40m59.068509269s) [noise=true , address=104.36.82.15:4258 , piece_id=776ZFSJQMWJI6WVFJ6O24X42S4TBIXQMVERNBSKDAYSBDIAAITCA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.346186427 (59h40m59.068749467s) [noise=true , address=104.36.82.12:4149 , piece_id=SQYE3VKF4ISMBZCOJXPEWFITEQWYQ4VYZOFUDYLH7KIOL34DJEAA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.346422309 (59h40m59.068985339s) [noise=true , address=45.91.116.4:4938 , piece_id=2NQNCSG2HOYBPHVDMPE4SXXGZJX2NO4UAOSLM6QWPXYSFO5NB43A, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.346652569 (59h40m59.069215609s) [noise=true , address=104.36.82.7:4909 , piece_id=QYXGSAB53RBNZ7GSWEBYLHSB2ZTYS4UVJHZ5LFHRFF2F7DYV4NSQ, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.346876579 (59h40m59.069439619s) [noise=true , address=45.91.116.6:4973 , piece_id=7BXFIPGIZXJW6WJ4XZ5BK5LDURPHVEKG2ELT7H6XXAIDPLKMLX3A, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:51.044906306 (59h41m0.767469346s) [upload=57260]: Next returned nil segment Jul 26 23:16:51.044926395 (59h41m0.767489435s) [upload=57260]: Waiting for error group managing segments... Jul 26 23:16:51.364934428 (59h41m1.087497458s) [noise=true , address=45.91.116.5:4298 , piece_id=X6DEY45NFWRHQPHUODFEB7WFWRKE6VIACDRJE2D2L4VYXHIWQEVA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.372466910 (59h41m1.09502995s) [noise=true , address=45.91.116.7:4009 , piece_id=AWFZDBRE4HGD3BOPUU7JITDWE4Y67W3CV7U4SH5BDFNQILUZA6SA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.459560226 (59h41m1.182123266s) [noise=true , address=45.91.116.8:4045 , piece_id=KJ3EZGGLW3DDDXUVWXAQIF5NIWUC2EVAPUG33TAW4D2YN75OKPOA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.463646744 (59h41m1.186209784s) [noise=true , address=45.91.116.6:4973 , piece_id=7BXFIPGIZXJW6WJ4XZ5BK5LDURPHVEKG2ELT7H6XXAIDPLKMLX3A, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.514342059 (59h41m1.23690509s) [noise=true , address=45.91.116.4:4938 , piece_id=2NQNCSG2HOYBPHVDMPE4SXXGZJX2NO4UAOSLM6QWPXYSFO5NB43A, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.522099993 (59h41m1.244663033s) [noise=true , address=45.91.116.12:4226 , piece_id=LYRTQQKREHKRKBVPFOR3AVIXQB6SVJT4ZECKKOH23WNRMZTGO4OA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.534967702 (59h41m1.257530732s) [noise=true , address=77.90.31.14:4661 , piece_id=4ZOGBM2UN4LVQMI26S3VSKNFQNRJE5UGL3AZEIRFPZZ7QL63QYIA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.551777084 (59h41m1.274340125s) [noise=true , address=45.91.116.10:4083 , piece_id=EK2S656KNAC7MSJGYK3FAL4ZJOBP4FF26VRPXHKPJGWJLM7JQFAQ, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.612576742 (59h41m1.335139772s) [noise=true , address=45.91.116.7:4011 , piece_id=MXU2WJATFMMHQUHMFCHPHPMI7QR2R5D67PHLO2M5KFN3V6VHC72A, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:17:18.232454490 (59h41m27.95501753s) [noise=true , address=77.90.31.10:4518 , piece_id=SJSEZPJQNAV4NXEMIOCNKZTW6RR7HXHWZ7C3BWJAEOJA3OSFPHTQ, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232501430 (59h41m27.95506446s) [noise=true , address=104.36.82.7:4909 , piece_id=QYXGSAB53RBNZ7GSWEBYLHSB2ZTYS4UVJHZ5LFHRFF2F7DYV4NSQ, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:40490->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232613455 (59h41m27.955176495s) [noise=true , address=104.36.82.12:4149 , piece_id=SQYE3VKF4ISMBZCOJXPEWFITEQWYQ4VYZOFUDYLH7KIOL34DJEAA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232642941 (59h41m27.955205981s) [noise=true , address=104.36.82.6:4876 , piece_id=IQB4KI5VC3UXWREOJLYV3FOEEPEZ42YTQI7W73Z37ZTQYCG2MK4A, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:57838->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232665414 (59h41m27.955228454s) [noise=true , address=104.36.82.10:4078 , piece_id=4L4Y4KPKVXESEYSZP6Q26R5J7V2CSXUNEI2SE7GB7B3A3OJBC4VQ, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232675724 (59h41m27.955238764s) [noise=true , address=104.36.82.15:4258 , piece_id=776ZFSJQMWJI6WVFJ6O24X42S4TBIXQMVERNBSKDAYSBDIAAITCA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232679040 (59h41m27.95524207s) [noise=true , address=104.36.82.10:4079 , piece_id=7IMEBZTMVUAAX2MY6HU3FSZJSDT72RHJY4FYP2RWVWHFRBDVJZAA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232698868 (59h41m27.955261908s) [seg_pos={0 0}, upload=57260]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:17:18.470668807 (59h41m28.193231837s) [upload=57260]: Done uploading segments... Jul 26 23:17:18.470692984 (59h41m28.193256024s) [upload=57260]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.240014192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X.X........X.X...........................XXXXXXXXXXXX] duration: 30.231120156s FAIL (2025-07-26 23:17:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.231154502s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [...............X..XX...X.X.X.X.......XX..XX.XX..XXX.........] duration: 49.676573527s FAIL (2025-07-26 23:16:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.676583997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [................X.........X...........X.......XXXXXXXXXXXXXX] duration: 30.230521731s FAIL (2025-07-26 23:16:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:37626->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123956809 (59h40m21.846519849s) [seg_pos={0 0}, upload=57245]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Jul 26 23:16:12.352465727 (59h40m22.075028757s) [upload=57245]: Done uploading segments... Jul 26 23:16:12.352489973 (59h40m22.075053013s) [upload=57245]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Jul 26 23:16:18.413235649 (59h40m28.135798679s) [upload=57250]: Starting upload Jul 26 23:16:18.413292178 (59h40m28.135855218s) [upload=57250]: Done starting upload Jul 26 23:16:18.413423529 (59h40m28.135986569s) [upload=57250]: Uploading segments... Jul 26 23:16:18.413528519 (59h40m28.136091559s) [upload=57250]: Got next segment. Inline: false Jul 26 23:16:18.413559760 (59h40m28.1361228s) [upload=57250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:16:18.413564689 (59h40m28.136127729s) [upload=57250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:16:18.667519300 (59h40m28.39008234s) [seg_pos={0 0}, upload=57250]: Begin upload segment... Jul 26 23:16:18.667557002 (59h40m28.390120042s) [seg_pos={0 0}, upload=57250]: Done begin upload segment. Jul 26 23:16:18.667579646 (59h40m28.390142686s) [noise=true , address=104.36.82.14:4222 , piece_id=OS4XGTOZTWAKYOMS2ZLZYIB5ZR2WBUH2BPS4H22S5G34HQFH4VIA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.667606818 (59h40m28.390169858s) [noise=true , address=45.91.116.5:4298 , piece_id=ZY5QZ77HXX3FJH5LRRJ66GRQERM3LR6SZJAMRHRJGBSPBRI5IBLQ, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.667971766 (59h40m28.390534806s) [noise=true , address=77.90.31.14:4662 , piece_id=NQBVGOYRRMLO4JOHOOLTORFQIFBVLX76Z5GU5YMW65AYN2LSXECA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.668036901 (59h40m28.390599931s) [noise=true , address=45.91.116.13:4154 , piece_id=ZHYOQB5TUIH55ZX5AZDDGOHRVFASBKWBSRKGC7GVL7GJPGFBFZVA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.668292620 (59h40m28.39085566s) [noise=true , address=45.91.116.12:4225 , piece_id=POW2EWW5MEZEZIPSHXIMTG57KQKSNZJGLIYVQTFL4G7KD4ZP2MPQ, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.668344710 (59h40m28.39090775s) [noise=true , address=45.91.116.7:4010 , piece_id=W73TWGFAVFDJPFVAQ5SISVSC4I5CL42I5VUMJESCD7KNRVF3EAGA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.668596962 (59h40m28.391160002s) [noise=true , address=104.36.82.12:4151 , piece_id=KZFN7CF7V3RNY4Z756QKNJSWUWTUY7JJL5JPQO5WXFAV7KSULJ7A, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.668613474 (59h40m28.391176514s) [noise=true , address=104.36.82.9:4006 , piece_id=TLCW3KCE2NJSCKPRSV5HEPBA3V5RWSHWB7ZVW7F7T4PDARP7DSPA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.668809188 (59h40m28.391372228s) [noise=true , address=104.36.82.2:4729 , piece_id=2ZHUXSOCB2JZBKOYKUH3Z4NJDOSRB2G4M26KFECENLLQE6TL5QNA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.669012006 (59h40m28.391575046s) [noise=true , address=45.91.116.5:4299 , piece_id=UMOUODFZPX2QMHKQCOPNUIZGK6GJGDOX73J7LKFJ3N72KPIY3FDA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.669265381 (59h40m28.391828421s) [noise=true , address=45.91.116.13:4155 , piece_id=HBWZE7EW7XISA6XBAKKG4EQRDZFGP4J762FTM5JGKVGQ2TOFX34A, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.669457508 (59h40m28.392020548s) [noise=true , address=45.91.116.10:4082 , piece_id=GO2JOK2OOJQ6MZZYMXC36HV5WUXBKVBPYGYYZBLA3SWCQFS7KLRA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.669637874 (59h40m28.392200904s) [noise=true , address=104.36.82.7:4909 , piece_id=3BOPO4RC2NWLA57IK4U45VOYTA7EGF7RLJYEQ4YIGL3UX3ONGZIQ, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.669856312 (59h40m28.392419352s) [noise=true , address=45.91.116.10:4081 , piece_id=FCP2BHYCYFXNH77SZQXWV3GQGXHLWUIHLHILINSDT2WA3X5P76EA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.670261177 (59h40m28.392824217s) [noise=true , address=45.91.116.6:4975 , piece_id=XTK3EKGBVXQVODGWN5Y2OIFHIKBFB4EHISRLF6P2X2MXD6IGJUMA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:18.670541292 (59h40m28.393104332s) [noise=true , address=45.91.116.4:4938 , piece_id=ELLJYEF3OG2JPJTBHBSBZIIMHNRDC7WENMC6KP2QOC5NH7VKNTDA, seg_pos={0 0}, upload=57250]: Uploading piece... Jul 26 23:16:20.423295495 (59h40m30.145858535s) [upload=57250]: Next returned nil segment Jul 26 23:16:20.423307698 (59h40m30.145870738s) [upload=57250]: Waiting for error group managing segments... Jul 26 23:16:20.701841838 (59h40m30.424404868s) [noise=true , address=77.90.31.14:4662 , piece_id=NQBVGOYRRMLO4JOHOOLTORFQIFBVLX76Z5GU5YMW65AYN2LSXECA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.751775958 (59h40m30.474338998s) [noise=true , address=45.91.116.12:4225 , piece_id=POW2EWW5MEZEZIPSHXIMTG57KQKSNZJGLIYVQTFL4G7KD4ZP2MPQ, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.763854988 (59h40m30.486418029s) [noise=true , address=45.91.116.6:4975 , piece_id=XTK3EKGBVXQVODGWN5Y2OIFHIKBFB4EHISRLF6P2X2MXD6IGJUMA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.770244895 (59h40m30.492807935s) [noise=true , address=45.91.116.13:4155 , piece_id=HBWZE7EW7XISA6XBAKKG4EQRDZFGP4J762FTM5JGKVGQ2TOFX34A, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.771829135 (59h40m30.494392175s) [noise=true , address=45.91.116.5:4299 , piece_id=UMOUODFZPX2QMHKQCOPNUIZGK6GJGDOX73J7LKFJ3N72KPIY3FDA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.772839769 (59h40m30.495402809s) [noise=true , address=45.91.116.13:4154 , piece_id=ZHYOQB5TUIH55ZX5AZDDGOHRVFASBKWBSRKGC7GVL7GJPGFBFZVA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.774328367 (59h40m30.496891397s) [noise=true , address=45.91.116.10:4082 , piece_id=GO2JOK2OOJQ6MZZYMXC36HV5WUXBKVBPYGYYZBLA3SWCQFS7KLRA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.832021041 (59h40m30.554584082s) [noise=true , address=45.91.116.7:4010 , piece_id=W73TWGFAVFDJPFVAQ5SISVSC4I5CL42I5VUMJESCD7KNRVF3EAGA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.856266520 (59h40m30.57882957s) [noise=true , address=45.91.116.10:4081 , piece_id=FCP2BHYCYFXNH77SZQXWV3GQGXHLWUIHLHILINSDT2WA3X5P76EA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:20.919662335 (59h40m30.642225375s) [noise=true , address=45.91.116.5:4298 , piece_id=ZY5QZ77HXX3FJH5LRRJ66GRQERM3LR6SZJAMRHRJGBSPBRI5IBLQ, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:21.002745639 (59h40m30.725308749s) [noise=true , address=45.91.116.4:4938 , piece_id=ELLJYEF3OG2JPJTBHBSBZIIMHNRDC7WENMC6KP2QOC5NH7VKNTDA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: <nil> Jul 26 23:16:47.768071644 (59h40m57.490634684s) [noise=true , address=104.36.82.14:4222 , piece_id=OS4XGTOZTWAKYOMS2ZLZYIB5ZR2WBUH2BPS4H22S5G34HQFH4VIA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:58214->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:47.768094548 (59h40m57.490657588s) [noise=true , address=104.36.82.9:4006 , piece_id=TLCW3KCE2NJSCKPRSV5HEPBA3V5RWSHWB7ZVW7F7T4PDARP7DSPA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:47.768123303 (59h40m57.490686343s) [noise=true , address=104.36.82.12:4151 , piece_id=KZFN7CF7V3RNY4Z756QKNJSWUWTUY7JJL5JPQO5WXFAV7KSULJ7A, seg_pos={0 0}, upload=57250]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:47.768134264 (59h40m57.490697364s) [noise=true , address=104.36.82.2:4729 , piece_id=2ZHUXSOCB2JZBKOYKUH3Z4NJDOSRB2G4M26KFECENLLQE6TL5QNA, seg_pos={0 0}, upload=57250]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:47.768166425 (59h40m57.490729465s) [noise=true , address=104.36.82.7:4909 , piece_id=3BOPO4RC2NWLA57IK4U45VOYTA7EGF7RLJYEQ4YIGL3UX3ONGZIQ, seg_pos={0 0}, upload=57250]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:42822->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:47.768185983 (59h40m57.490749013s) [seg_pos={0 0}, upload=57250]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:16:47.996956281 (59h40m57.719519311s) [upload=57250]: Done uploading segments... Jul 26 23:16:47.996981249 (59h40m57.719544289s) [upload=57250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.230539444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.880588008s FAIL (2025-07-26 23:16:38 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.880613377s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23815343s FAIL (2025-07-26 23:16:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== VR7F5FTXEBMRET5LJAM7ZMQKO3IERQOUHGOXGA, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.584301144 (59h39m48.306864184s) [noise=true , address=104.36.82.45:4905 , piece_id=FHW3XITLXXLTLTX6A7AUIJD55ZCVX4NVF4SFIBHNMFOE5E4D5KEA, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.584556643 (59h39m48.307119683s) [noise=true , address=64.49.13.137:4502 , piece_id=ULN5VOQHKR4Q7VW4CODOXB2N3BPL4EWOPOV6R6NCGGKSREAQDFOQ, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.584815619 (59h39m48.307378659s) [noise=true , address=104.36.82.56:4256 , piece_id=HTTUMC6ZMD2QZIPYFAW4NX3URBH6M2Q33LTFVOIH4NKUKHMRSKGA, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.584852269 (59h39m48.307415309s) [noise=true , address=77.90.31.10:4540 , piece_id=6MS222B7C7OXDJUXW2JI4VUYJ5MYTLXG7VH7DJLVSCCA3WOWZZPA, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.585149437 (59h39m48.307712467s) [noise=true , address=79.127.205.245:4223 , piece_id=TR5UXHF74GG2Y3WKTWSZ4T2ZZFLAJBKIXTMDZPBUG3ZMMEOULXKA, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.585161490 (59h39m48.30772453s) [noise=true , address=79.127.205.235:4685 , piece_id=XE5MUOX4YAASV5SWVEL4XAFQU5YSJMEVSDN57RJET5O2YOOSIXBA, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.585469098 (59h39m48.308032128s) [noise=true , address=104.36.82.54:4191 , piece_id=5JGLVXCD3NXSGPPCI44RY4LJSKQHVUDMA5HNNL4YQWXAR3NI5P3A, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.585473577 (59h39m48.308036607s) [noise=true , address=79.127.205.246:4288 , piece_id=K25SNXCRNVOSPRY5BZYVN6QJFFIXJVGBJVRYX4HBHAQCP2X7XK6A, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.585682727 (59h39m48.308245767s) [noise=true , address=45.91.116.7:4015 , piece_id=43GVTLBDJX4FTZKMQN7EJNQCU7SP33PLN3GX4WX3RS762CRG2ZLQ, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.585747741 (59h39m48.308310771s) [noise=true , address=104.36.82.55:4209 , piece_id=LAJAMNGIR7MAABTUPP7M4X24HWG6ONETOS4BFU5KFTD5L4Y5DGRA, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.585925021 (59h39m48.308488061s) [noise=true , address=79.127.205.234:4615 , piece_id=GZHAHAMXR76VTKSHNL5EGBJEBTVXRGEJWXQALVZ2D7JBF2DDEXQQ, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.586014562 (59h39m48.308577602s) [noise=true , address=104.36.82.46:4502 , piece_id=3UWBDVCMWIFBSDDKR46PVG7QQR3AJTJWQEPPKPASTQCQFIXMGT5A, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.586228592 (59h39m48.308791632s) [noise=true , address=104.36.82.52:4100 , piece_id=PA5GJTRERCAZSRVTE27W7KYBSCTY2DQOC36Y32A4ZJVQMG7SYA3Q, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.586298485 (59h39m48.308861525s) [noise=true , address=45.91.116.11:4121 , piece_id=V7HOLGUC5SAA72RPUMD2ERFC6VB57IHAC53CNA5M7TV3IS6644HQ, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.586469092 (59h39m48.309032132s) [noise=true , address=104.36.82.47:4552 , piece_id=EYP2ZT6BPAIVBKL4BJUQCCFFFENPYK4QP6S7W2JOOYYOHFQU2DDA, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.586554906 (59h39m48.309117946s) [noise=true , address=77.90.31.15:4722 , piece_id=7NXVA2UPR6IOELID6PJ6MFIRUZJUYRMV3YKBZ2P36YDAHV32AWZQ, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:38.586687260 (59h39m48.3092503s) [noise=true , address=64.49.13.139:4478 , piece_id=2BFJDP5LD6OMXY7TUNT2P4JDOMDQC7PGXZE2C7SP2N6HLNJ3BXCQ, seg_pos={0 0}, upload=57242]: Uploading piece... Jul 26 23:15:39.682110066 (59h39m49.404673096s) [upload=57242]: Next returned nil segment Jul 26 23:15:39.682131107 (59h39m49.404694147s) [upload=57242]: Waiting for error group managing segments... Jul 26 23:15:39.994356719 (59h39m49.716919759s) [noise=true , address=79.127.205.233:4645 , piece_id=FIGDRNYKZ4KJD7VR7F5FTXEBMRET5LJAM7ZMQKO3IERQOUHGOXGA, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:15:39.994652365 (59h39m49.717215395s) [noise=true , address=79.127.205.246:4288 , piece_id=K25SNXCRNVOSPRY5BZYVN6QJFFIXJVGBJVRYX4HBHAQCP2X7XK6A, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:15:39.997501957 (59h39m49.720064987s) [noise=true , address=79.127.205.245:4223 , piece_id=TR5UXHF74GG2Y3WKTWSZ4T2ZZFLAJBKIXTMDZPBUG3ZMMEOULXKA, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:15:40.005724008 (59h39m49.728287058s) [noise=true , address=79.127.205.235:4685 , piece_id=XE5MUOX4YAASV5SWVEL4XAFQU5YSJMEVSDN57RJET5O2YOOSIXBA, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:15:40.174650666 (59h39m49.897213706s) [noise=true , address=64.49.13.137:4502 , piece_id=ULN5VOQHKR4Q7VW4CODOXB2N3BPL4EWOPOV6R6NCGGKSREAQDFOQ, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 26 23:15:40.310470942 (59h39m50.033033982s) [noise=true , address=79.127.205.234:4615 , piece_id=GZHAHAMXR76VTKSHNL5EGBJEBTVXRGEJWXQALVZ2D7JBF2DDEXQQ, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:15:40.527160686 (59h39m50.249723726s) [noise=true , address=64.49.13.139:4478 , piece_id=2BFJDP5LD6OMXY7TUNT2P4JDOMDQC7PGXZE2C7SP2N6HLNJ3BXCQ, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:15:40.570842943 (59h39m50.293405983s) [noise=true , address=45.91.116.7:4015 , piece_id=43GVTLBDJX4FTZKMQN7EJNQCU7SP33PLN3GX4WX3RS762CRG2ZLQ, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:15:40.686684433 (59h39m50.409247473s) [noise=true , address=77.90.31.15:4722 , piece_id=7NXVA2UPR6IOELID6PJ6MFIRUZJUYRMV3YKBZ2P36YDAHV32AWZQ, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:15:45.175238782 (59h39m54.897801822s) [noise=true , address=45.91.116.11:4121 , piece_id=V7HOLGUC5SAA72RPUMD2ERFC6VB57IHAC53CNA5M7TV3IS6644HQ, seg_pos={0 0}, upload=57242]: Done uploading piece. err: <nil> Jul 26 23:16:37.488656793 (59h40m47.211219833s) [noise=true , address=104.36.82.56:4256 , piece_id=HTTUMC6ZMD2QZIPYFAW4NX3URBH6M2Q33LTFVOIH4NKUKHMRSKGA, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:37.488708922 (59h40m47.211271963s) [noise=true , address=104.36.82.47:4552 , piece_id=EYP2ZT6BPAIVBKL4BJUQCCFFFENPYK4QP6S7W2JOOYYOHFQU2DDA, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:37.488745222 (59h40m47.211308262s) [noise=true , address=77.90.31.10:4540 , piece_id=6MS222B7C7OXDJUXW2JI4VUYJ5MYTLXG7VH7DJLVSCCA3WOWZZPA, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: write tcp 188.166.225.170:45372->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:37.488968929 (59h40m47.21153197s) [noise=true , address=104.36.82.45:4905 , piece_id=FHW3XITLXXLTLTX6A7AUIJD55ZCVX4NVF4SFIBHNMFOE5E4D5KEA, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: rpc: dial tcp 104.36.82.45:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:37.489036620 (59h40m47.21159966s) [noise=true , address=104.36.82.54:4191 , piece_id=5JGLVXCD3NXSGPPCI44RY4LJSKQHVUDMA5HNNL4YQWXAR3NI5P3A, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:16:37.489118276 (59h40m47.211681306s) [noise=true , address=104.36.82.52:4100 , piece_id=PA5GJTRERCAZSRVTE27W7KYBSCTY2DQOC36Y32A4ZJVQMG7SYA3Q, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:104.36.82.52:4100): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:16:37.489167921 (59h40m47.211730961s) [noise=true , address=104.36.82.46:4502 , piece_id=3UWBDVCMWIFBSDDKR46PVG7QQR3AJTJWQEPPKPASTQCQFIXMGT5A, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: write tcp 188.166.225.170:50542->104.36.82.46:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:37.489199872 (59h40m47.211762912s) [noise=true , address=104.36.82.55:4209 , piece_id=LAJAMNGIR7MAABTUPP7M4X24HWG6ONETOS4BFU5KFTD5L4Y5DGRA, seg_pos={0 0}, upload=57242]: Done uploading piece. err: ecclient: upload failed (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR, address:104.36.82.55:4209): protocol: write tcp 188.166.225.170:54848->104.36.82.55:4209: use of closed network connection; conn closed: op Jul 26 23:16:37.489222255 (59h40m47.211785295s) [seg_pos={0 0}, upload=57242]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:16:37.724646082 (59h40m47.447209112s) [upload=57242]: Done uploading segments... Jul 26 23:16:37.724672563 (59h40m47.447235593s) [upload=57242]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.238165053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................XXX.X..X....] duration: 23.981850334s FAIL (2025-07-26 23:16:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.981873859s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.................X.........X...........X.......XXXXXXXXXXXXX] duration: 30.231457095s FAIL (2025-07-26 23:16:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ce_id=CGWURPG3Y654SJJVJRMRCRMMHBGTQXBBQXFKTWDURFGXDSJRRVRA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.003925030 (59h39m52.72648806s) [noise=true , address=104.36.82.7:4909 , piece_id=TJBHC57EOSYLEN5MUIFH7DTG3WRXKBOHOCFWJDPSXVMMGIJCKK6A, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.004253879 (59h39m52.726816919s) [noise=true , address=104.36.82.10:4078 , piece_id=YASI4H7R2L4MJIOWJTTZ5HZHM523UCNQTY6GSBXF4MDJ6UJCBVKA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.004360343 (59h39m52.726923383s) [noise=true , address=104.36.82.3:4767 , piece_id=NJ5WIM65ECNKUZ5HGDPPR6OPRCAFTJXI233ZZQ47DSA7GQFVF42A, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.004641891 (59h39m52.727204931s) [noise=true , address=104.36.82.15:4257 , piece_id=LSIAY2IND5TF2GUIGKTSGMU3Y72G46FJY2CYIZ6MSTZJRK3C7ETA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.004660707 (59h39m52.727223747s) [noise=true , address=104.36.82.9:4007 , piece_id=7PDXGHOXHTDATXTMJ53QTMML6452XBFI2OWRENMN2DP635U5IWAQ, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.004938849 (59h39m52.72750188s) [noise=true , address=104.36.82.14:4223 , piece_id=I4XWEZ2PDAJ4CKY65J4CQHHZHV5U3FDYBNHPSOUOHUJ3YUHJARKA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.005014695 (59h39m52.727577735s) [noise=true , address=45.91.116.5:4297 , piece_id=PBXEESBQGPC4LIHOJLQQYXN2VPKMDV3MC4JYLKAMF7YRSFMSK7VA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.005206572 (59h39m52.727769602s) [noise=true , address=45.91.116.11:4117 , piece_id=TZAD33UAGWBLQP5DAAUJIX443XIDJECLNWZX6OLPC26P6JK2ZXMQ, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.005280192 (59h39m52.727843232s) [noise=true , address=104.36.82.8:4972 , piece_id=2FUKZL4JGEBK5LK5EDHZZQLTK5QPSW2Z36A5KGIAO5TEYGMONZUA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.005463664 (59h39m52.728026704s) [noise=true , address=45.91.116.7:4010 , piece_id=QR34N6JHLQI4ZY5OILYQJFAO7M73C5NF3XCRGWPSTRCKXJJ3VWGA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.005616366 (59h39m52.728179406s) [noise=true , address=104.36.82.2:4730 , piece_id=BFXE4MPLG6QGDSYUUGU2REUOIX6ONDMEOG5GWYKUA3AW2WM7O56A, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.005736355 (59h39m52.728299395s) [noise=true , address=104.36.82.2:4732 , piece_id=7BCMKKGMHNTM4DXTJMAZVORKPLDREFCXIRA73RVCE6ZDSDV2PTKA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.005885801 (59h39m52.728448841s) [noise=true , address=104.36.82.11:4115 , piece_id=FFV2CZZCAECNMUNH53DAL2S4UCGAAVNNV66QPX52RB2HQNGQHG2A, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.006085434 (59h39m52.728648474s) [noise=true , address=104.36.82.10:4079 , piece_id=4AAOWXM7MRWUAB4LIMPM4XODJU6RPXE5UUJNVV36UXYDCYFHBFJA, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:43.006283031 (59h39m52.728846071s) [noise=true , address=45.91.116.6:4974 , piece_id=LJWJNXLWRCFH3KNP7FAUA3GZYMILK24RXNS7BQ5LHXAQKHNYU25A, seg_pos={0 0}, upload=57245]: Uploading piece... Jul 26 23:15:44.748214295 (59h39m54.470777335s) [upload=57245]: Next returned nil segment Jul 26 23:15:44.748232871 (59h39m54.470795901s) [upload=57245]: Waiting for error group managing segments... Jul 26 23:15:45.065186509 (59h39m54.787749549s) [noise=true , address=45.91.116.11:4117 , piece_id=TZAD33UAGWBLQP5DAAUJIX443XIDJECLNWZX6OLPC26P6JK2ZXMQ, seg_pos={0 0}, upload=57245]: Done uploading piece. err: <nil> Jul 26 23:15:45.114348420 (59h39m54.83691146s) [noise=true , address=45.91.116.6:4974 , piece_id=LJWJNXLWRCFH3KNP7FAUA3GZYMILK24RXNS7BQ5LHXAQKHNYU25A, seg_pos={0 0}, upload=57245]: Done uploading piece. err: <nil> Jul 26 23:15:45.114928541 (59h39m54.837491581s) [noise=true , address=45.91.116.7:4010 , piece_id=QR34N6JHLQI4ZY5OILYQJFAO7M73C5NF3XCRGWPSTRCKXJJ3VWGA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: <nil> Jul 26 23:15:45.171955471 (59h39m54.894518511s) [noise=true , address=45.91.116.5:4297 , piece_id=PBXEESBQGPC4LIHOJLQQYXN2VPKMDV3MC4JYLKAMF7YRSFMSK7VA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: <nil> Jul 26 23:16:12.123339899 (59h40m21.845902939s) [noise=true , address=104.36.82.8:4972 , piece_id=2FUKZL4JGEBK5LK5EDHZZQLTK5QPSW2Z36A5KGIAO5TEYGMONZUA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123482031 (59h40m21.846045071s) [noise=true , address=104.36.82.2:4730 , piece_id=BFXE4MPLG6QGDSYUUGU2REUOIX6ONDMEOG5GWYKUA3AW2WM7O56A, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:16:12.123527669 (59h40m21.846090709s) [noise=true , address=104.36.82.9:4007 , piece_id=7PDXGHOXHTDATXTMJ53QTMML6452XBFI2OWRENMN2DP635U5IWAQ, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123558818 (59h40m21.846121858s) [noise=true , address=104.36.82.14:4223 , piece_id=I4XWEZ2PDAJ4CKY65J4CQHHZHV5U3FDYBNHPSOUOHUJ3YUHJARKA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123588825 (59h40m21.846151865s) [noise=true , address=104.36.82.2:4732 , piece_id=7BCMKKGMHNTM4DXTJMAZVORKPLDREFCXIRA73RVCE6ZDSDV2PTKA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123624123 (59h40m21.846187163s) [noise=true , address=104.36.82.3:4767 , piece_id=NJ5WIM65ECNKUZ5HGDPPR6OPRCAFTJXI233ZZQ47DSA7GQFVF42A, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123786413 (59h40m21.846349443s) [noise=true , address=104.36.82.10:4079 , piece_id=4AAOWXM7MRWUAB4LIMPM4XODJU6RPXE5UUJNVV36UXYDCYFHBFJA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123837070 (59h40m21.84640012s) [noise=true , address=104.36.82.15:4257 , piece_id=LSIAY2IND5TF2GUIGKTSGMU3Y72G46FJY2CYIZ6MSTZJRK3C7ETA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123862990 (59h40m21.84642603s) [noise=true , address=104.36.82.7:4909 , piece_id=TJBHC57EOSYLEN5MUIFH7DTG3WRXKBOHOCFWJDPSXVMMGIJCKK6A, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:46896->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123891645 (59h40m21.846454675s) [noise=true , address=104.36.82.10:4078 , piece_id=YASI4H7R2L4MJIOWJTTZ5HZHM523UCNQTY6GSBXF4MDJ6UJCBVKA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:52238->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123911613 (59h40m21.846474653s) [noise=true , address=104.36.82.11:4115 , piece_id=FFV2CZZCAECNMUNH53DAL2S4UCGAAVNNV66QPX52RB2HQNGQHG2A, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123937963 (59h40m21.846501003s) [noise=true , address=104.36.82.6:4874 , piece_id=CGWURPG3Y654SJJVJRMRCRMMHBGTQXBBQXFKTWDURFGXDSJRRVRA, seg_pos={0 0}, upload=57245]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:37626->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:12.123956809 (59h40m21.846519849s) [seg_pos={0 0}, upload=57245]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Jul 26 23:16:12.352465727 (59h40m22.075028757s) [upload=57245]: Done uploading segments... Jul 26 23:16:12.352489973 (59h40m22.075053013s) [upload=57245]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.231476191s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X.X........X.X...........................XXXXXXXXXXX] duration: 30.236482669s FAIL (2025-07-26 23:16:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.236500133s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X...............X..XX...X.X.X.X.......XX..XX.XX..XXX........] duration: 43.654431751s FAIL (2025-07-26 23:16:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.654454104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23395401s FAIL (2025-07-26 23:15:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== piece... Jul 26 23:14:38.113756575 (59h38m47.836319615s) [noise=true , address=77.90.31.15:4729 , piece_id=P74DKFYEGJGCFPRHT6RVPROVQWTPU63SSIVB45APDK2YL2YB5UVA, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.113868368 (59h38m47.836431398s) [noise=true , address=104.36.82.56:4248 , piece_id=A23HX3B5KZ3K7X72BSRILI4PGIHH4R76EHA5NXO5GSCKWQELEO4Q, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.114198831 (59h38m47.836761871s) [noise=true , address=45.91.116.11:4151 , piece_id=QXR6OY4W6SMPEA35XCAQ4DCBYBAVPZXOUL3QXCDS5EF3IZCJMP6Q, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.114258806 (59h38m47.836821846s) [noise=true , address=45.91.116.9:4270 , piece_id=2B3MQK2XZEBPMLFAIMMMLA2CTUY5YYUOXOWNSCO7IPNOP53IIDKA, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.114462706 (59h38m47.837025746s) [noise=true , address=104.36.82.53:4150 , piece_id=YVCWKPWWFTTWQH2OZCF6RA6TPMJZEHZ3WINSW6ZQFP6WQXWJ5J3Q, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.114566946 (59h38m47.837129986s) [noise=true , address=104.36.82.13:4201 , piece_id=Q7SCJOOUZEBWTUCJ37VVZBRYV2IRKANPWJYU7IE7Z7NF6L6VW66Q, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.114730669 (59h38m47.837293709s) [noise=true , address=79.127.205.245:4218 , piece_id=EVFQ43ITJ4OJHMPFX33ZDHKUMMDWAMI7BIIZNJJYOE5UEGKXCV6Q, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.114868943 (59h38m47.837431984s) [noise=true , address=77.90.31.19:4847 , piece_id=A6BBKIQU76RTRXHRRQKP64PPUJVX6AKGF55ZY3XWXIOAXUBZAVXA, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.114982110 (59h38m47.83754515s) [noise=true , address=79.127.219.38:4001 , piece_id=QQ2SRBW4XRHNV4JQPO42K5Z34EEDVQ4O2NTSJIUODTCPL3HLKQMQ, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.115219775 (59h38m47.837782815s) [noise=true , address=104.36.82.47:4523 , piece_id=L4TJEA4DAVT5UVU5HEDZM7MME4AVSJQW6ZA7WUOKTWL4TWEWLXEQ, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.115277686 (59h38m47.837840726s) [noise=true , address=104.36.82.55:4196 , piece_id=2ZCLTPUFSTP6N5AVPNHFLLBQPTELTEBDWHCUW47SUSVHQGFNXB5A, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.115421951 (59h38m47.837984991s) [noise=true , address=79.127.201.215:4346 , piece_id=NI6SU3KWH3356MUIYQ4XNARGZGP52HH6CRZBIDNIXAFNPJUXIREA, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.115569674 (59h38m47.838132744s) [noise=true , address=79.127.205.233:4633 , piece_id=A3M4IAKWAKZWDFJVL3TWN43AJHLQU773BIYK5SYAWSW6NJRKNZOQ, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.115628187 (59h38m47.838191227s) [noise=true , address=104.36.82.52:4112 , piece_id=BQNPW5P2ITVQ4UHN4EJMFJ5VYTJPKYUS3ZU7JJMN7G6CNBGJGLLQ, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.115855311 (59h38m47.838418341s) [noise=true , address=77.90.31.16:4761 , piece_id=VA7GHP5IL5IHFLCNJ2FURJPXQ3ISJEK3DXWAZ3SV5AFQHKVIBK2Q, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.115920345 (59h38m47.838483375s) [noise=true , address=104.36.82.46:4494 , piece_id=XDVXVVJFKPT32VK7MU4WSR3NBPHSMFDNKZA52TRDFCJHWSV7UUUQ, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:38.116053810 (59h38m47.83861684s) [noise=true , address=79.127.219.48:4090 , piece_id=ZNYAR7ISCOTANV4CT253IEBAYT2B5BHWBQPOBXETXQ67JNITOREA, seg_pos={0 0}, upload=57226]: Uploading piece... Jul 26 23:14:39.198924793 (59h38m48.921487823s) [noise=true , address=79.127.219.48:4090 , piece_id=ZNYAR7ISCOTANV4CT253IEBAYT2B5BHWBQPOBXETXQ67JNITOREA, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 23:14:39.211407416 (59h38m48.933970456s) [upload=57226]: Next returned nil segment Jul 26 23:14:39.211420721 (59h38m48.933983761s) [upload=57226]: Waiting for error group managing segments... Jul 26 23:14:39.514156927 (59h38m49.236719957s) [noise=true , address=79.127.205.245:4218 , piece_id=EVFQ43ITJ4OJHMPFX33ZDHKUMMDWAMI7BIIZNJJYOE5UEGKXCV6Q, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:14:39.521048142 (59h38m49.243611182s) [noise=true , address=79.127.205.233:4633 , piece_id=A3M4IAKWAKZWDFJVL3TWN43AJHLQU773BIYK5SYAWSW6NJRKNZOQ, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:14:40.078133306 (59h38m49.800696346s) [noise=true , address=45.91.116.11:4151 , piece_id=QXR6OY4W6SMPEA35XCAQ4DCBYBAVPZXOUL3QXCDS5EF3IZCJMP6Q, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:14:40.223670001 (59h38m49.946233041s) [noise=true , address=45.91.116.9:4270 , piece_id=2B3MQK2XZEBPMLFAIMMMLA2CTUY5YYUOXOWNSCO7IPNOP53IIDKA, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:14:40.231857236 (59h38m49.954420276s) [noise=true , address=79.127.219.38:4001 , piece_id=QQ2SRBW4XRHNV4JQPO42K5Z34EEDVQ4O2NTSJIUODTCPL3HLKQMQ, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:14:40.910583186 (59h38m50.633146226s) [noise=true , address=77.90.31.16:4761 , piece_id=VA7GHP5IL5IHFLCNJ2FURJPXQ3ISJEK3DXWAZ3SV5AFQHKVIBK2Q, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:14:41.070282368 (59h38m50.792845399s) [noise=true , address=77.90.31.15:4729 , piece_id=P74DKFYEGJGCFPRHT6RVPROVQWTPU63SSIVB45APDK2YL2YB5UVA, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:14:53.688385993 (59h39m3.410949033s) [noise=true , address=77.90.31.19:4847 , piece_id=A6BBKIQU76RTRXHRRQKP64PPUJVX6AKGF55ZY3XWXIOAXUBZAVXA, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:15:12.282481409 (59h39m22.005044449s) [noise=true , address=79.127.201.215:4346 , piece_id=NI6SU3KWH3356MUIYQ4XNARGZGP52HH6CRZBIDNIXAFNPJUXIREA, seg_pos={0 0}, upload=57226]: Done uploading piece. err: <nil> Jul 26 23:15:37.019235047 (59h39m46.741798087s) [noise=true , address=104.36.82.53:4150 , piece_id=YVCWKPWWFTTWQH2OZCF6RA6TPMJZEHZ3WINSW6ZQFP6WQXWJ5J3Q, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: rpc: dial tcp 104.36.82.53:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:37.019275463 (59h39m46.741838504s) [noise=true , address=104.36.82.55:4196 , piece_id=2ZCLTPUFSTP6N5AVPNHFLLBQPTELTEBDWHCUW47SUSVHQGFNXB5A, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: rpc: dial tcp 104.36.82.55:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:37.019301253 (59h39m46.741864293s) [noise=true , address=104.36.82.52:4112 , piece_id=BQNPW5P2ITVQ4UHN4EJMFJ5VYTJPKYUS3ZU7JJMN7G6CNBGJGLLQ, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: rpc: dial tcp 104.36.82.52:4112: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:37.019541673 (59h39m46.742104723s) [noise=true , address=104.36.82.47:4523 , piece_id=L4TJEA4DAVT5UVU5HEDZM7MME4AVSJQW6ZA7WUOKTWL4TWEWLXEQ, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: rpc: dial tcp 104.36.82.47:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:37.019579004 (59h39m46.742142044s) [noise=true , address=104.36.82.54:4177 , piece_id=D7FV2WSD47V3LZVDOYGTGMORRO2TV3GG2J5IFF6JCTPPLWVNOTGA, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: rpc: dial tcp 104.36.82.54:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:37.019619502 (59h39m46.742182542s) [noise=true , address=104.36.82.56:4248 , piece_id=A23HX3B5KZ3K7X72BSRILI4PGIHH4R76EHA5NXO5GSCKWQELEO4Q, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: rpc: dial tcp 104.36.82.56:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:37.019624261 (59h39m46.742187291s) [noise=true , address=104.36.82.46:4494 , piece_id=XDVXVVJFKPT32VK7MU4WSR3NBPHSMFDNKZA52TRDFCJHWSV7UUUQ, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: rpc: dial tcp 104.36.82.46:4494: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:37.019653968 (59h39m46.742216998s) [noise=true , address=104.36.82.13:4201 , piece_id=Q7SCJOOUZEBWTUCJ37VVZBRYV2IRKANPWJYU7IE7Z7NF6L6VW66Q, seg_pos={0 0}, upload=57226]: Done uploading piece. err: ecclient: upload failed (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6, address:104.36.82.13:4201): protocol: write tcp 188.166.225.170:41506->104.36.82.13:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:37.019692201 (59h39m46.742255241s) [seg_pos={0 0}, upload=57226]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:15:37.251391763 (59h39m46.973954803s) [upload=57226]: Done uploading segments... Jul 26 23:15:37.251415889 (59h39m46.973978929s) [upload=57226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.233966663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.854001249s FAIL (2025-07-26 23:15:31 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.854028069s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................X.........X...........X.......XXXXXXXXXXXX] duration: 30.230863654s FAIL (2025-07-26 23:15:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== segments... Jul 26 23:14:52.431451825 (59h39m2.154014855s) [upload=57231]: Got next segment. Inline: false Jul 26 23:14:52.431486793 (59h39m2.154049823s) [upload=57231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:14:52.431491532 (59h39m2.154054562s) [upload=57231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:14:52.666126720 (59h39m2.38868976s) [seg_pos={0 0}, upload=57231]: Begin upload segment... Jul 26 23:14:52.666180072 (59h39m2.388743102s) [seg_pos={0 0}, upload=57231]: Done begin upload segment. Jul 26 23:14:52.666212936 (59h39m2.388775976s) [noise=true , address=45.91.116.6:4974 , piece_id=OTONOISSFEUZOKU3GLU6HUGOK567XOO2KU6GBVV23PYJODQUIDHA, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.666220279 (59h39m2.388783319s) [noise=true , address=104.36.82.3:4765 , piece_id=MFL5SIN7QRJHHW6FTJ6GCTIWQXBTZYPFPBJZXHLYTZUWFV32JG6Q, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.666579046 (59h39m2.389142086s) [noise=true , address=45.91.116.5:4299 , piece_id=QLO7JHQUEPKKEQ4DFM2DQPWZKGLUTKOOAZVH47NTRO2LIQURJQEQ, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.666729684 (59h39m2.389292724s) [noise=true , address=45.91.116.8:4047 , piece_id=YIGJDYGEKT3NSXHP7EB4VCXZQJJALAOSLQM3A44K7U5HHF6MGCFQ, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.666923524 (59h39m2.389486564s) [noise=true , address=104.36.82.9:4006 , piece_id=OLSYCIEMLXIICUDE2U5BJOT2ACNNQXDMVDYIBCTFE4XUGXUGLA3A, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.667219821 (59h39m2.389782882s) [noise=true , address=104.36.82.16:4293 , piece_id=EVRUL3DM4YJD7XOBZS4URZE7UA4THRFZ5A5RSUIHGBBXACYQO4LA, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.667246442 (59h39m2.389809483s) [noise=true , address=104.36.82.2:4729 , piece_id=PJOBKDSA4HPLDO2FCWIOORMWHP35QGKFP6A5AP62Q73PABSEOFBQ, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.667519174 (59h39m2.390082214s) [noise=true , address=45.91.116.7:4009 , piece_id=CY5FBD5OYJVUU4GHDLSAJRA4ZHKCMGKNABZAMX5X4ZTCXYV5H73Q, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.667809029 (59h39m2.390372069s) [noise=true , address=104.36.82.14:4222 , piece_id=VHO3QNYND4RZPAY3EOCPAP47YAKGXWL4OL3K3CLNYJPINWYRPICQ, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.668062855 (59h39m2.390625895s) [noise=true , address=45.91.116.4:4937 , piece_id=OLSTCVJV7ZPQRMCYDAOQRBZ44DJX6GEMVMYM5ZM5Z4J77WM7LPAA, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.668380893 (59h39m2.390943933s) [noise=true , address=104.36.82.6:4875 , piece_id=RTAEOF25LZSQU3OTAB7I6FRYB64DCPLPT2PYXR4ZITVXLFINNY2A, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.668617366 (59h39m2.391180406s) [noise=true , address=104.36.82.2:4730 , piece_id=GIGW7BWEU4VFORQUAGWCL37SYZQJAMY6OIAZZAMKDLMKI4GO7Y2A, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.668822267 (59h39m2.391385307s) [noise=true , address=45.91.116.5:4297 , piece_id=T354K2I4ZQSUIF3HU3VAQBEL44BPKJRVWUR6FW3IJSY5B6I7V6MQ, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.669107443 (59h39m2.391670483s) [noise=true , address=45.91.116.12:4227 , piece_id=5VJBNIMM4HTXGAPMQBBPAASGUQ7XGYG2VGKELD36P4OYIYKLD4YQ, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.669428808 (59h39m2.391991848s) [noise=true , address=104.36.82.10:4077 , piece_id=4GL4WQPDUOUNL633TP4ONBMHO5EIX24JYGQ7RFFR6YMKSMY4GTEQ, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:52.670369388 (59h39m2.392932418s) [noise=true , address=77.90.31.10:4520 , piece_id=AEVRLA2J6E73FDBFZACNQCFJKDDYPRSABOASVXN3NWHE3ASQ6LYA, seg_pos={0 0}, upload=57231]: Uploading piece... Jul 26 23:14:54.397493402 (59h39m4.120056442s) [upload=57231]: Next returned nil segment Jul 26 23:14:54.397506216 (59h39m4.120069246s) [upload=57231]: Waiting for error group managing segments... Jul 26 23:14:54.674889064 (59h39m4.397452104s) [noise=true , address=45.91.116.8:4047 , piece_id=YIGJDYGEKT3NSXHP7EB4VCXZQJJALAOSLQM3A44K7U5HHF6MGCFQ, seg_pos={0 0}, upload=57231]: Done uploading piece. err: <nil> Jul 26 23:14:54.684542765 (59h39m4.407105805s) [noise=true , address=45.91.116.5:4299 , piece_id=QLO7JHQUEPKKEQ4DFM2DQPWZKGLUTKOOAZVH47NTRO2LIQURJQEQ, seg_pos={0 0}, upload=57231]: Done uploading piece. err: <nil> Jul 26 23:14:54.769376948 (59h39m4.491939988s) [noise=true , address=45.91.116.12:4227 , piece_id=5VJBNIMM4HTXGAPMQBBPAASGUQ7XGYG2VGKELD36P4OYIYKLD4YQ, seg_pos={0 0}, upload=57231]: Done uploading piece. err: <nil> Jul 26 23:14:54.786021284 (59h39m4.508584314s) [noise=true , address=45.91.116.7:4009 , piece_id=CY5FBD5OYJVUU4GHDLSAJRA4ZHKCMGKNABZAMX5X4ZTCXYV5H73Q, seg_pos={0 0}, upload=57231]: Done uploading piece. err: <nil> Jul 26 23:14:54.788960918 (59h39m4.511523948s) [noise=true , address=45.91.116.5:4297 , piece_id=T354K2I4ZQSUIF3HU3VAQBEL44BPKJRVWUR6FW3IJSY5B6I7V6MQ, seg_pos={0 0}, upload=57231]: Done uploading piece. err: <nil> Jul 26 23:14:54.841468120 (59h39m4.56403115s) [noise=true , address=45.91.116.6:4974 , piece_id=OTONOISSFEUZOKU3GLU6HUGOK567XOO2KU6GBVV23PYJODQUIDHA, seg_pos={0 0}, upload=57231]: Done uploading piece. err: <nil> Jul 26 23:14:54.889028107 (59h39m4.611591147s) [noise=true , address=45.91.116.4:4937 , piece_id=OLSTCVJV7ZPQRMCYDAOQRBZ44DJX6GEMVMYM5ZM5Z4J77WM7LPAA, seg_pos={0 0}, upload=57231]: Done uploading piece. err: <nil> Jul 26 23:15:21.793640851 (59h39m31.516203891s) [noise=true , address=104.36.82.14:4222 , piece_id=VHO3QNYND4RZPAY3EOCPAP47YAKGXWL4OL3K3CLNYJPINWYRPICQ, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793687641 (59h39m31.516250681s) [noise=true , address=104.36.82.2:4730 , piece_id=GIGW7BWEU4VFORQUAGWCL37SYZQJAMY6OIAZZAMKDLMKI4GO7Y2A, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793724832 (59h39m31.516287872s) [noise=true , address=104.36.82.10:4077 , piece_id=4GL4WQPDUOUNL633TP4ONBMHO5EIX24JYGQ7RFFR6YMKSMY4GTEQ, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:55272->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793737757 (59h39m31.516300787s) [noise=true , address=104.36.82.9:4006 , piece_id=OLSYCIEMLXIICUDE2U5BJOT2ACNNQXDMVDYIBCTFE4XUGXUGLA3A, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:60402->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793763326 (59h39m31.516326366s) [noise=true , address=104.36.82.6:4875 , piece_id=RTAEOF25LZSQU3OTAB7I6FRYB64DCPLPT2PYXR4ZITVXLFINNY2A, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793791259 (59h39m31.516354289s) [noise=true , address=104.36.82.2:4729 , piece_id=PJOBKDSA4HPLDO2FCWIOORMWHP35QGKFP6A5AP62Q73PABSEOFBQ, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:41038->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793824743 (59h39m31.516387773s) [noise=true , address=77.90.31.10:4520 , piece_id=AEVRLA2J6E73FDBFZACNQCFJKDDYPRSABOASVXN3NWHE3ASQ6LYA, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793835754 (59h39m31.516398794s) [noise=true , address=104.36.82.16:4293 , piece_id=EVRUL3DM4YJD7XOBZS4URZE7UA4THRFZ5A5RSUIHGBBXACYQO4LA, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793842538 (59h39m31.516405568s) [noise=true , address=104.36.82.3:4765 , piece_id=MFL5SIN7QRJHHW6FTJ6GCTIWQXBTZYPFPBJZXHLYTZUWFV32JG6Q, seg_pos={0 0}, upload=57231]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:21.793862916 (59h39m31.516425956s) [seg_pos={0 0}, upload=57231]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:15:22.022855558 (59h39m31.745418588s) [upload=57231]: Done uploading segments... Jul 26 23:15:22.022881057 (59h39m31.745444097s) [upload=57231]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.23088279s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XXX.X..X...] duration: 9.194771038s FAIL (2025-07-26 23:15:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.194783792s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........X.X........X.X...........................XXXXXXXXXX] duration: 30.241102176s FAIL (2025-07-26 23:15:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.241115902s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXX.X..X..] duration: 17.170936869s FAIL (2025-07-26 23:15:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.170953722s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.048838072s FAIL (2025-07-26 23:14:52 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.048868511s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXX] duration: 1m0.237444937s FAIL (2025-07-26 23:14:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 2H5XWI4OE7HVRHV3JOUXA, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.647580206 (59h37m47.370143246s) [noise=true , address=104.36.82.54:4169 , piece_id=MYFS2YTJEVHBTT6HEIMP7EWIFR35NTKKAT4LGDTSG2YW3FC4FJHA, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.647989618 (59h37m47.370552658s) [noise=true , address=79.127.205.235:4675 , piece_id=5ONDMLZHLOMHNFKPWBBWSUF5B6Q2PS2ZNQAK3WXPZR3BLTYTOPMQ, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.648086184 (59h37m47.370649224s) [noise=true , address=77.90.31.18:4821 , piece_id=PUXMFE2IFLY4RCZ3NMJOEDBGNY3DYBVUN42NOHTTXTYERI4DR2GQ, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.648485267 (59h37m47.371048307s) [noise=true , address=77.90.31.19:4850 , piece_id=WYYGMOVBXXKZX6T6PKZQH6IIBPTKC3ITQ5ZKOMPDNTF4LBB2CVDQ, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.648568606 (59h37m47.371131646s) [noise=true , address=104.36.82.56:4235 , piece_id=CV665LQKWFBBVZLPONZWWLBH3YVIYAAXXEKECW5BCR5I7LCTBAIA, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.648885282 (59h37m47.371448323s) [noise=true , address=104.36.82.53:4153 , piece_id=H36TYMPOZMQHJ3Q7UR6LBDZZD72YSHL6AJT72HBVZ5RV4ZV7IJKQ, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.649719278 (59h37m47.372282308s) [noise=true , address=104.36.82.55:4207 , piece_id=KOQKTSUJAEPNEHQNIY47AIO6UTM6FX5SBCATLXID7QO27DJSBWSA, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.650124864 (59h37m47.372687914s) [noise=true , address=104.36.82.42:4804 , piece_id=7YD6N5ET33GU276ONH5YDUIJVF6LYBBFQ2TUYEUVARQQ2RVXDI7A, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.650631112 (59h37m47.373194152s) [noise=true , address=45.91.116.9:4277 , piece_id=GM7MO4BBDLT2KZYKC2WDYRPCHEE3EYDQWZWXMDWOEVPYTJ5VEDKQ, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.650844591 (59h37m47.373407631s) [noise=true , address=77.90.31.10:4536 , piece_id=XQECCG6NGUTUHX3R4BJCFLJDF4CHKCCP2WEV7YWQ4O4UP3N4H4NA, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.651104138 (59h37m47.373667178s) [noise=true , address=104.36.82.52:4094 , piece_id=ZBNHKBZOAFKPMFKONVSGY7FTHC545JMH5XWYZY47MXSHCKK7HX3A, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.651334979 (59h37m47.373898019s) [noise=true , address=79.127.201.215:4348 , piece_id=SFTWSNA5MECXFDKR7SS3F7TIM54YCOEP6BZOFXFGAQVPGBZQ637A, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.651657306 (59h37m47.374220346s) [noise=true , address=104.36.82.20:4465 , piece_id=PXA757M7TJK3ZKIGZGEQGPY6W3J5T7AIVYOFFZ6PJN2QENSIIJYA, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.651685520 (59h37m47.37424856s) [noise=true , address=64.49.13.170:4721 , piece_id=AEI6DAKBBKN253B5LTWZT5PRQNWZQESQGN67AQXWYO5JSTXIJPQQ, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.652000633 (59h37m47.374563663s) [noise=true , address=79.127.205.234:4611 , piece_id=EZLJAHTXTFMUJ3WLSUWCAT2WBSDQ5BH5ZJWVVDHOCMSGT6KE3ROA, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:37.652298222 (59h37m47.374861262s) [noise=true , address=104.36.82.46:4481 , piece_id=5PXHPVZFCNSGQDXKEOU34CBIA6KO5U6VUDBKKZPCIUXGIOASIZOA, seg_pos={0 0}, upload=57212]: Uploading piece... Jul 26 23:13:38.761362380 (59h37m48.48392541s) [upload=57212]: Next returned nil segment Jul 26 23:13:38.761376066 (59h37m48.483939106s) [upload=57212]: Waiting for error group managing segments... Jul 26 23:13:39.066997793 (59h37m48.789560823s) [noise=true , address=79.127.205.234:4611 , piece_id=EZLJAHTXTFMUJ3WLSUWCAT2WBSDQ5BH5ZJWVVDHOCMSGT6KE3ROA, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:13:39.068352806 (59h37m48.790915846s) [noise=true , address=79.127.205.235:4675 , piece_id=5ONDMLZHLOMHNFKPWBBWSUF5B6Q2PS2ZNQAK3WXPZR3BLTYTOPMQ, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:13:39.602172355 (59h37m49.324735395s) [noise=true , address=79.127.201.215:4348 , piece_id=SFTWSNA5MECXFDKR7SS3F7TIM54YCOEP6BZOFXFGAQVPGBZQ637A, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:13:39.629655399 (59h37m49.352218439s) [noise=true , address=104.36.82.42:4804 , piece_id=7YD6N5ET33GU276ONH5YDUIJVF6LYBBFQ2TUYEUVARQQ2RVXDI7A, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:13:39.633977248 (59h37m49.356540288s) [noise=true , address=77.90.31.11:4584 , piece_id=T7DDBLEQNRJN7HJTVCUPUWXYLHR36RT2H5XWI4OE7HVRHV3JOUXA, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:13:39.677892641 (59h37m49.400455671s) [noise=true , address=77.90.31.18:4821 , piece_id=PUXMFE2IFLY4RCZ3NMJOEDBGNY3DYBVUN42NOHTTXTYERI4DR2GQ, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:13:43.771781138 (59h37m53.494344178s) [noise=true , address=45.91.116.9:4277 , piece_id=GM7MO4BBDLT2KZYKC2WDYRPCHEE3EYDQWZWXMDWOEVPYTJ5VEDKQ, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:13:48.400510146 (59h37m58.123073186s) [noise=true , address=64.49.13.170:4721 , piece_id=AEI6DAKBBKN253B5LTWZT5PRQNWZQESQGN67AQXWYO5JSTXIJPQQ, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:14:15.488256773 (59h38m25.210819813s) [noise=true , address=77.90.31.19:4850 , piece_id=WYYGMOVBXXKZX6T6PKZQH6IIBPTKC3ITQ5ZKOMPDNTF4LBB2CVDQ, seg_pos={0 0}, upload=57212]: Done uploading piece. err: <nil> Jul 26 23:14:36.556916185 (59h38m46.279479225s) [noise=true , address=104.36.82.52:4094 , piece_id=ZBNHKBZOAFKPMFKONVSGY7FTHC545JMH5XWYZY47MXSHCKK7HX3A, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: rpc: dial tcp 104.36.82.52:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.556993804 (59h38m46.279556844s) [noise=true , address=104.36.82.55:4207 , piece_id=KOQKTSUJAEPNEHQNIY47AIO6UTM6FX5SBCATLXID7QO27DJSBWSA, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.557025294 (59h38m46.279588334s) [noise=true , address=104.36.82.20:4465 , piece_id=PXA757M7TJK3ZKIGZGEQGPY6W3J5T7AIVYOFFZ6PJN2QENSIIJYA, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:1aGm17yvKPgD3nqcQyoMVvgM3gqXDDrKPNzXJY1xf2uBdcfobS, address:104.36.82.20:4465): protocol: rpc: dial tcp 104.36.82.20:4465: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.557057606 (59h38m46.279620646s) [noise=true , address=104.36.82.46:4481 , piece_id=5PXHPVZFCNSGQDXKEOU34CBIA6KO5U6VUDBKKZPCIUXGIOASIZOA, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 188.166.225.170:36626->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.557089126 (59h38m46.279652166s) [noise=true , address=104.36.82.47:4542 , piece_id=JDUL4SOPCADK6TQHN4EJ2FIUKF3TEBZSCGR6K7QSRIQZWKG2UGGQ, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: write tcp 188.166.225.170:45036->104.36.82.47:4542: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.557203896 (59h38m46.279766936s) [noise=true , address=104.36.82.54:4169 , piece_id=MYFS2YTJEVHBTT6HEIMP7EWIFR35NTKKAT4LGDTSG2YW3FC4FJHA, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.557234735 (59h38m46.279797775s) [noise=true , address=104.36.82.53:4153 , piece_id=H36TYMPOZMQHJ3Q7UR6LBDZZD72YSHL6AJT72HBVZ5RV4ZV7IJKQ, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:37448->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.557260775 (59h38m46.279823815s) [noise=true , address=77.90.31.10:4536 , piece_id=XQECCG6NGUTUHX3R4BJCFLJDF4CHKCCP2WEV7YWQ4O4UP3N4H4NA, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.557285471 (59h38m46.279848511s) [noise=true , address=104.36.82.56:4235 , piece_id=CV665LQKWFBBVZLPONZWWLBH3YVIYAAXXEKECW5BCR5I7LCTBAIA, seg_pos={0 0}, upload=57212]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: rpc: dial tcp 104.36.82.56:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.557308776 (59h38m46.279871816s) [seg_pos={0 0}, upload=57212]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:14:36.792430185 (59h38m46.514993225s) [upload=57212]: Done uploading segments... Jul 26 23:14:36.792457497 (59h38m46.515020537s) [upload=57212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.237461849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXX.X..X.] duration: 28.121504667s FAIL (2025-07-26 23:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.121524968s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........X.X........X.X...........................XXXXXXXXX] duration: 30.342897869s FAIL (2025-07-26 23:14:33 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.34293487s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...................X.........X...........X.......XXXXXXXXXXX] duration: 30.239468093s FAIL (2025-07-26 23:14:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== bjectBegin Jul 26 23:14:01.846568425 (59h38m11.569131475s) [upload=57216]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:14:02.089248786 (59h38m11.811811836s) [seg_pos={0 0}, upload=57216]: Begin upload segment... Jul 26 23:14:02.089296588 (59h38m11.811859638s) [seg_pos={0 0}, upload=57216]: Done begin upload segment. Jul 26 23:14:02.089323559 (59h38m11.811886619s) [noise=true , address=104.36.82.12:4150 , piece_id=EYUTIUZNEVNWCXZTHY2VEXNN35GPBEF33JXUWIMLIKFOED24HQPA, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.089377032 (59h38m11.811940082s) [noise=true , address=45.91.116.14:4191 , piece_id=HLK27LNFIEQQQQ2HESBNBY67JISRXGV3ROS33EAK3QQM7GEUMW5Q, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.089771857 (59h38m11.812334907s) [noise=true , address=104.36.82.4:4802 , piece_id=HCCIB2PI3YHL3WL7MAMLZDA6ASUHT3MUG56W45OTXYRFFTBTS3LA, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.089854535 (59h38m11.812417585s) [noise=true , address=45.91.116.13:4155 , piece_id=ZA7OCHRDXDJDQMDPNSYBLNGQ57AL2UUEBRSFNDZRPPRWAXJC42CQ, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.090110544 (59h38m11.812673585s) [noise=true , address=45.91.116.10:4081 , piece_id=NGQ2FATZNHVTHQGWQEUXA4BG6VJ5NCV2L3HMHNZRWM76BC3FQ7JQ, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.090144350 (59h38m11.81270739s) [noise=true , address=104.36.82.4:4801 , piece_id=XS7XB4AMALUEITRCV3BMFMCFCKYGNXR4AO5ODG22YQMIPOZ6KXGQ, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.090357828 (59h38m11.812920868s) [noise=true , address=104.36.82.9:4007 , piece_id=MNBRHZKISX7PTYJXEFA2LGP7KVLLN5DD6QOWCK7MHIQRW6RYFYRA, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.090447901 (59h38m11.813010941s) [noise=true , address=104.36.82.15:4259 , piece_id=6ABVVSWMESOCS26SMVSPE7RMS7ZPODAFEYQ4BNNSJ62UQS7QFPVQ, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.090605191 (59h38m11.813168241s) [noise=true , address=104.36.82.9:4006 , piece_id=JK44MKTLY6GABPHLHUBUUULUANHLQBDYBA4HBNDHJGNJNUGUQ4TA, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.090789354 (59h38m11.813352404s) [noise=true , address=45.91.116.5:4297 , piece_id=2ICGXB2F327RMBORS433VYAFTSR3MN42FAL7BKDZZDZGTC2BY7RA, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.090858526 (59h38m11.813421566s) [noise=true , address=104.36.82.8:4969 , piece_id=5K6NJQR3ZMP2DORIATNST7CX25W4EKM6AK2RMDUJFRGG4WGTK4WA, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.091121860 (59h38m11.8136849s) [noise=true , address=104.36.82.6:4874 , piece_id=N76IDUFVIJENQ23XXW2J4FXKF5JDWHPQJQ2JRA7QCLOK2B2UWOVQ, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.091429037 (59h38m11.813992087s) [noise=true , address=104.36.82.14:4223 , piece_id=ZNHHCEOVBJJF6AC27ULVC52IAJIP5KQ6XANCDM7A3BJSTL264CLA, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.091739742 (59h38m11.814302782s) [noise=true , address=77.90.31.19:4841 , piece_id=JPTU3R4C7VTM7IR72TXNJRNRDSLOZUMVC5CQ62ASMCVTA7QXWV6A, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.092049755 (59h38m11.814612795s) [noise=true , address=104.36.82.2:4729 , piece_id=TY2WX474OJLNR6P7OWX6ICNIZEUAM4PLMH252TH67YEVK7PTMG4A, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:02.092417058 (59h38m11.814980108s) [noise=true , address=104.36.82.16:4294 , piece_id=SUUVZEZFAXYZ24CCHML4YEA2DS4QUO4Y3BPV5CCWP5P7KGBKLBZQ, seg_pos={0 0}, upload=57216]: Uploading piece... Jul 26 23:14:03.857087520 (59h38m13.57965056s) [upload=57216]: Next returned nil segment Jul 26 23:14:03.857104453 (59h38m13.579667483s) [upload=57216]: Waiting for error group managing segments... Jul 26 23:14:04.140462863 (59h38m13.863025903s) [noise=true , address=104.36.82.8:4969 , piece_id=5K6NJQR3ZMP2DORIATNST7CX25W4EKM6AK2RMDUJFRGG4WGTK4WA, seg_pos={0 0}, upload=57216]: Done uploading piece. err: <nil> Jul 26 23:14:04.188131037 (59h38m13.910694077s) [noise=true , address=45.91.116.13:4155 , piece_id=ZA7OCHRDXDJDQMDPNSYBLNGQ57AL2UUEBRSFNDZRPPRWAXJC42CQ, seg_pos={0 0}, upload=57216]: Done uploading piece. err: <nil> Jul 26 23:14:04.194229235 (59h38m13.916792295s) [noise=true , address=45.91.116.10:4081 , piece_id=NGQ2FATZNHVTHQGWQEUXA4BG6VJ5NCV2L3HMHNZRWM76BC3FQ7JQ, seg_pos={0 0}, upload=57216]: Done uploading piece. err: <nil> Jul 26 23:14:04.199879315 (59h38m13.922442355s) [noise=true , address=45.91.116.5:4297 , piece_id=2ICGXB2F327RMBORS433VYAFTSR3MN42FAL7BKDZZDZGTC2BY7RA, seg_pos={0 0}, upload=57216]: Done uploading piece. err: <nil> Jul 26 23:14:04.205953486 (59h38m13.928516526s) [noise=true , address=45.91.116.14:4191 , piece_id=HLK27LNFIEQQQQ2HESBNBY67JISRXGV3ROS33EAK3QQM7GEUMW5Q, seg_pos={0 0}, upload=57216]: Done uploading piece. err: <nil> Jul 26 23:14:05.981310265 (59h38m15.703873295s) [noise=true , address=77.90.31.19:4841 , piece_id=JPTU3R4C7VTM7IR72TXNJRNRDSLOZUMVC5CQ62ASMCVTA7QXWV6A, seg_pos={0 0}, upload=57216]: Done uploading piece. err: <nil> Jul 26 23:14:31.197384738 (59h38m40.919947778s) [noise=true , address=104.36.82.16:4294 , piece_id=SUUVZEZFAXYZ24CCHML4YEA2DS4QUO4Y3BPV5CCWP5P7KGBKLBZQ, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197440224 (59h38m40.920003264s) [noise=true , address=104.36.82.9:4007 , piece_id=MNBRHZKISX7PTYJXEFA2LGP7KVLLN5DD6QOWCK7MHIQRW6RYFYRA, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:53808->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197473357 (59h38m40.920036397s) [noise=true , address=104.36.82.15:4259 , piece_id=6ABVVSWMESOCS26SMVSPE7RMS7ZPODAFEYQ4BNNSJ62UQS7QFPVQ, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197619747 (59h38m40.920182817s) [noise=true , address=104.36.82.6:4874 , piece_id=N76IDUFVIJENQ23XXW2J4FXKF5JDWHPQJQ2JRA7QCLOK2B2UWOVQ, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197706033 (59h38m40.920269073s) [noise=true , address=104.36.82.14:4223 , piece_id=ZNHHCEOVBJJF6AC27ULVC52IAJIP5KQ6XANCDM7A3BJSTL264CLA, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197752822 (59h38m40.920315872s) [noise=true , address=104.36.82.2:4729 , piece_id=TY2WX474OJLNR6P7OWX6ICNIZEUAM4PLMH252TH67YEVK7PTMG4A, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197889954 (59h38m40.920453004s) [noise=true , address=104.36.82.4:4802 , piece_id=HCCIB2PI3YHL3WL7MAMLZDA6ASUHT3MUG56W45OTXYRFFTBTS3LA, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197933828 (59h38m40.920496878s) [noise=true , address=104.36.82.9:4006 , piece_id=JK44MKTLY6GABPHLHUBUUULUANHLQBDYBA4HBNDHJGNJNUGUQ4TA, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197969817 (59h38m40.920532857s) [noise=true , address=104.36.82.12:4150 , piece_id=EYUTIUZNEVNWCXZTHY2VEXNN35GPBEF33JXUWIMLIKFOED24HQPA, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.197987742 (59h38m40.920550792s) [noise=true , address=104.36.82.4:4801 , piece_id=XS7XB4AMALUEITRCV3BMFMCFCKYGNXR4AO5ODG22YQMIPOZ6KXGQ, seg_pos={0 0}, upload=57216]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:43660->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:31.198026666 (59h38m40.920589706s) [seg_pos={0 0}, upload=57216]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:14:31.435112113 (59h38m41.157675143s) [upload=57216]: Done uploading segments... Jul 26 23:14:31.435137991 (59h38m41.157701031s) [upload=57216]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.239489455s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X...............X..XX...X.X.X.X.......XX..XX.XX..XXX.......] duration: 17.029278227s FAIL (2025-07-26 23:14:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.029300149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....................X.........X...........X.......XXXXXXXXXX] duration: 30.234547302s FAIL (2025-07-26 23:13:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== pload=57192]: Done uploading segments... Jul 26 23:13:04.033396890 (59h37m13.75595994s) [upload=57192]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:13:28.047555465 (59h37m37.770118505s) [upload=57207]: Starting upload Jul 26 23:13:28.047651629 (59h37m37.770214679s) [upload=57207]: Done starting upload Jul 26 23:13:28.047731833 (59h37m37.770294873s) [upload=57207]: Uploading segments... Jul 26 23:13:28.047898341 (59h37m37.770461382s) [upload=57207]: Got next segment. Inline: false Jul 26 23:13:28.047930192 (59h37m37.770493222s) [upload=57207]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:13:28.047935382 (59h37m37.770498422s) [upload=57207]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:13:28.277609860 (59h37m38.0001729s) [seg_pos={0 0}, upload=57207]: Begin upload segment... Jul 26 23:13:28.277655919 (59h37m38.000218959s) [seg_pos={0 0}, upload=57207]: Done begin upload segment. Jul 26 23:13:28.277676377 (59h37m38.000239417s) [noise=true , address=45.91.116.8:4047 , piece_id=2JDHF37FULKM5IETHEHRUEL4HAK2XNSYZFRK3CEMBXASMPXKPVTQ, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.277714050 (59h37m38.00027708s) [noise=true , address=45.91.116.10:4082 , piece_id=ZXI237SACOK7MP4M4IQJQ53RKI3C7XTKTRZRAQ56GGXTECTXMO7Q, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.278091411 (59h37m38.000654451s) [noise=true , address=77.90.31.14:4661 , piece_id=DADHE6K7GRJ5DDLRV5RJVQW4CB3GXGDXCJPOHKQGH4USFWHZUI2A, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.278135446 (59h37m38.000698486s) [noise=true , address=45.91.116.4:4938 , piece_id=OA2QYWEIA2FQVZO4GXDVEVCWK7OV3RTFTV3YBXQI3OFZHJO3LADQ, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.278453905 (59h37m38.001016945s) [noise=true , address=104.36.82.2:4731 , piece_id=Z67BJB7SI3QD3UQ4FZ6TXP3FVYV76M6Y7LAV6Q3MVBHEUWCDAWUQ, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.278723871 (59h37m38.001286911s) [noise=true , address=45.91.116.14:4189 , piece_id=ZW6AASXSJVNRWMEHCKA4B76FNMVAV33JZI4CNKJE4Q5ROTW3GX3A, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.278945656 (59h37m38.001508696s) [noise=true , address=104.36.82.15:4257 , piece_id=UBPHKYWNNJQIWL5YEWFKGHRL4EFMOJESOS47AYC24HIRO4XJ5I3A, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.279161098 (59h37m38.001724128s) [noise=true , address=104.36.82.4:4802 , piece_id=UL7E25EVSNB4DHKB3XAVZLOPBT2NCFEJJ45MDKJXLEC3JHIZJPYQ, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.279376029 (59h37m38.001939069s) [noise=true , address=45.91.116.6:4975 , piece_id=FXOSRRYSX2T55XC3QPHX3MN23JLGSHMLFYOL24DK3A7IJMWOBIKQ, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.279598735 (59h37m38.002161775s) [noise=true , address=45.91.116.7:4010 , piece_id=TJG4I4CLXKWR5P4D7XPPRSWDEARXQUCPV63N5W3FJMABGLEHUYIA, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.279809318 (59h37m38.002372359s) [noise=true , address=77.90.31.14:4662 , piece_id=V3Y7DH7O3YEDK6HHF4GSGPKACTSWUNR6NBI25YNGE4J7COWENW5Q, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.279986387 (59h37m38.002549427s) [noise=true , address=77.90.31.17:4770 , piece_id=FSWJAT2VVYUVVJSIWNY5KTFFO5AZHCXEUJQI5MVFMDFUAKSIFX6A, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.280186440 (59h37m38.00274947s) [noise=true , address=104.36.82.9:4006 , piece_id=HBPPMWNGXJ37A264YUYYYRCK7I3PGOY7YG7JGS6E6HNRCEXVLNLA, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.280422501 (59h37m38.002985541s) [noise=true , address=104.36.82.9:4005 , piece_id=CHXTEP3EHAF4AT4YL4BNWNIOYYNYJRC4NR4LMAHFLKK4TLD5ZXLQ, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.280606043 (59h37m38.003169083s) [noise=true , address=77.90.31.11:4554 , piece_id=BEMF4RGAY4YAMN7SYLOZGA3NTBKF6X5UUFSVOY2EUIBCMIRL4QHQ, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:28.280865509 (59h37m38.003428539s) [noise=true , address=104.36.82.16:4295 , piece_id=7XVLOIF2B5VKEBWKPAVHEJKDTVPHGXPB53IY4N6X7MUEPZCILVDQ, seg_pos={0 0}, upload=57207]: Uploading piece... Jul 26 23:13:30.021045714 (59h37m39.743608754s) [upload=57207]: Next returned nil segment Jul 26 23:13:30.021067876 (59h37m39.743630916s) [upload=57207]: Waiting for error group managing segments... Jul 26 23:13:30.297055124 (59h37m40.019618164s) [noise=true , address=45.91.116.4:4938 , piece_id=OA2QYWEIA2FQVZO4GXDVEVCWK7OV3RTFTV3YBXQI3OFZHJO3LADQ, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:30.388933205 (59h37m40.111496245s) [noise=true , address=45.91.116.10:4082 , piece_id=ZXI237SACOK7MP4M4IQJQ53RKI3C7XTKTRZRAQ56GGXTECTXMO7Q, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:30.388957611 (59h37m40.111520651s) [noise=true , address=77.90.31.11:4554 , piece_id=BEMF4RGAY4YAMN7SYLOZGA3NTBKF6X5UUFSVOY2EUIBCMIRL4QHQ, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:30.389360001 (59h37m40.111923031s) [noise=true , address=45.91.116.14:4189 , piece_id=ZW6AASXSJVNRWMEHCKA4B76FNMVAV33JZI4CNKJE4Q5ROTW3GX3A, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:30.389682739 (59h37m40.112245779s) [noise=true , address=77.90.31.14:4661 , piece_id=DADHE6K7GRJ5DDLRV5RJVQW4CB3GXGDXCJPOHKQGH4USFWHZUI2A, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:30.392691074 (59h37m40.115254134s) [noise=true , address=45.91.116.8:4047 , piece_id=2JDHF37FULKM5IETHEHRUEL4HAK2XNSYZFRK3CEMBXASMPXKPVTQ, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:30.436630333 (59h37m40.159193373s) [noise=true , address=45.91.116.6:4975 , piece_id=FXOSRRYSX2T55XC3QPHX3MN23JLGSHMLFYOL24DK3A7IJMWOBIKQ, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:30.460726918 (59h37m40.183289958s) [noise=true , address=77.90.31.14:4662 , piece_id=V3Y7DH7O3YEDK6HHF4GSGPKACTSWUNR6NBI25YNGE4J7COWENW5Q, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:30.461259567 (59h37m40.183822607s) [noise=true , address=45.91.116.7:4010 , piece_id=TJG4I4CLXKWR5P4D7XPPRSWDEARXQUCPV63N5W3FJMABGLEHUYIA, seg_pos={0 0}, upload=57207]: Done uploading piece. err: <nil> Jul 26 23:13:57.402238092 (59h38m7.124801132s) [noise=true , address=104.36.82.4:4802 , piece_id=UL7E25EVSNB4DHKB3XAVZLOPBT2NCFEJJ45MDKJXLEC3JHIZJPYQ, seg_pos={0 0}, upload=57207]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:57.402251197 (59h38m7.124814238s) [noise=true , address=104.36.82.2:4731 , piece_id=Z67BJB7SI3QD3UQ4FZ6TXP3FVYV76M6Y7LAV6Q3MVBHEUWCDAWUQ, seg_pos={0 0}, upload=57207]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:57.402292556 (59h38m7.124855596s) [noise=true , address=104.36.82.9:4006 , piece_id=HBPPMWNGXJ37A264YUYYYRCK7I3PGOY7YG7JGS6E6HNRCEXVLNLA, seg_pos={0 0}, upload=57207]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:57.402301844 (59h38m7.124864884s) [noise=true , address=77.90.31.17:4770 , piece_id=FSWJAT2VVYUVVJSIWNY5KTFFO5AZHCXEUJQI5MVFMDFUAKSIFX6A, seg_pos={0 0}, upload=57207]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:53774->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:57.402343945 (59h38m7.124906985s) [noise=true , address=104.36.82.9:4005 , piece_id=CHXTEP3EHAF4AT4YL4BNWNIOYYNYJRC4NR4LMAHFLKK4TLD5ZXLQ, seg_pos={0 0}, upload=57207]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:57.402347141 (59h38m7.124910181s) [noise=true , address=104.36.82.15:4257 , piece_id=UBPHKYWNNJQIWL5YEWFKGHRL4EFMOJESOS47AYC24HIRO4XJ5I3A, seg_pos={0 0}, upload=57207]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:33454->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:57.402380154 (59h38m7.124943194s) [noise=true , address=104.36.82.16:4295 , piece_id=7XVLOIF2B5VKEBWKPAVHEJKDTVPHGXPB53IY4N6X7MUEPZCILVDQ, seg_pos={0 0}, upload=57207]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:37392->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:57.402408367 (59h38m7.124971408s) [seg_pos={0 0}, upload=57207]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:13:57.634732112 (59h38m7.357295152s) [upload=57207]: Done uploading segments... Jul 26 23:13:57.634757000 (59h38m7.35732004s) [upload=57207]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.234574495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..X...............X..XX...X.X.X.X.......XX..XX.XX..XXX......] duration: 41.343305289s FAIL (2025-07-26 23:13:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.343329006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.534443552s FAIL (2025-07-26 23:13:38 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.534471284s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXX] duration: 1m0.227536657s FAIL (2025-07-26 23:13:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 6m46.903743892s) [noise=true , address=104.36.82.54:4168 , piece_id=YHOHRO5S7VLDIW3HW2RVEPE2ANRASG25WKEHT47UPXWO5MOA22CA, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.181273359 (59h36m46.903836399s) [noise=true , address=104.36.82.55:4197 , piece_id=HDR3KAUPCN7WV462SEOPJMFAU2FLLBE4JQY622ZZWA7Y5J5WNUKA, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.181430499 (59h36m46.903993539s) [noise=true , address=104.36.82.47:4552 , piece_id=BQF777JA3H26YZFZ5MARKY4SXTRPHLT5HY7NRHSIYKUKLOYTDEXA, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.181531954 (59h36m46.904094994s) [noise=true , address=79.127.205.245:4246 , piece_id=7KCISIC5QQFNARGGA3WSVC6B3L26OYI4CLV73O4O2BW52NO5UWIQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.181674987 (59h36m46.904238027s) [noise=true , address=104.36.82.53:4135 , piece_id=RRFL56VSJOR6CH5V7X5B7T37XS3M4RE52ZRNQTCXZT7IQYBY2DYA, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.181814234 (59h36m46.904377274s) [noise=true , address=104.36.82.56:4249 , piece_id=T374HUIJB52XONPQQ33B3N74BUHDPX52RKCMYW4H72JDE6F6AAPQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.181947539 (59h36m46.904510579s) [noise=true , address=79.127.205.235:4679 , piece_id=5IQTHVY5FUUX7UBNHPNG2UA3ZAASZ4AQRH5KSHECLNXIRGVYHQDQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.182080473 (59h36m46.904643513s) [noise=true , address=77.90.31.10:4533 , piece_id=FEVPHEKTNEXEECHDBWNA6527KTLNX64IZXIVZXGZ65WWE64G7G4Q, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.182267872 (59h36m46.904830912s) [noise=true , address=64.49.13.155:4916 , piece_id=QJHZCLP4JL5H6R6UKJJDNZZ4TYKE4KPU2KCFQBUPA76MTW5R4MBQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.182340862 (59h36m46.904903902s) [noise=true , address=64.49.13.155:4923 , piece_id=7JZJD4NQL2HKE6F4DLX7FTZQB2HWK5UF42U7Y43XXDH4FPE3O2CQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.182541526 (59h36m46.905104556s) [noise=true , address=104.36.82.15:4266 , piece_id=TVOAVS32BMXXZUO3FQHFYUXJ2MQCIQOIYMJ4AMFT2S7T32O5OSSQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.182591211 (59h36m46.905154251s) [noise=true , address=77.90.31.15:4721 , piece_id=OAP7NKIKFWFWIRWJWQL5QT5OBCCQX55STOQBAHNYZSUU4DNLXFMQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.182749524 (59h36m46.905312554s) [noise=true , address=79.127.201.214:4321 , piece_id=6TIYGXIEDHRT4TCLTEZ4XNJC5XAYAJ3IRSUNJBTPZ2DHH4QNV7SA, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.182794459 (59h36m46.905357499s) [noise=true , address=79.127.205.234:4591 , piece_id=SOPDKDLLKGTD3NMYS6DFNYBS36ANDZUA2HZOGQQRSIZJX6ORQ7AA, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.182991536 (59h36m46.905554576s) [noise=true , address=77.90.31.16:4766 , piece_id=WKSRDTYA54FXRLQE7G4GZPMFFJGR2ZQFHDSIQUBPPFEIQRJZ4YSQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:37.183008559 (59h36m46.905571599s) [noise=true , address=104.36.82.52:4105 , piece_id=R6DZELQO36N3NPB2PETSUD374OTSA3LVCGQTDZM7GYLJE465F7CQ, seg_pos={0 0}, upload=57194]: Uploading piece... Jul 26 23:12:38.028928412 (59h36m47.751491452s) [noise=true , address=64.49.13.155:4916 , piece_id=QJHZCLP4JL5H6R6UKJJDNZZ4TYKE4KPU2KCFQBUPA76MTW5R4MBQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B, address:64.49.13.155:4916): protocol: storage node overloaded, request limit: 2000 Jul 26 23:12:38.276012378 (59h36m47.998575408s) [upload=57194]: Next returned nil segment Jul 26 23:12:38.276028640 (59h36m47.99859168s) [upload=57194]: Waiting for error group managing segments... Jul 26 23:12:38.566010773 (59h36m48.288573813s) [noise=true , address=79.127.205.245:4246 , piece_id=7KCISIC5QQFNARGGA3WSVC6B3L26OYI4CLV73O4O2BW52NO5UWIQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: <nil> Jul 26 23:12:38.597666862 (59h36m48.320229902s) [noise=true , address=79.127.205.235:4679 , piece_id=5IQTHVY5FUUX7UBNHPNG2UA3ZAASZ4AQRH5KSHECLNXIRGVYHQDQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: <nil> Jul 26 23:12:38.600838099 (59h36m48.323401139s) [noise=true , address=79.127.205.234:4591 , piece_id=SOPDKDLLKGTD3NMYS6DFNYBS36ANDZUA2HZOGQQRSIZJX6ORQ7AA, seg_pos={0 0}, upload=57194]: Done uploading piece. err: <nil> Jul 26 23:12:38.607956889 (59h36m48.330519929s) [noise=true , address=79.127.205.246:4285 , piece_id=ORB7FKNHK4QCKAZOEQHEAJ2J2W253DRAPB3AH3WJDZV6W67T55VQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: <nil> Jul 26 23:12:44.141989823 (59h36m53.864552863s) [noise=true , address=64.49.13.155:4923 , piece_id=7JZJD4NQL2HKE6F4DLX7FTZQB2HWK5UF42U7Y43XXDH4FPE3O2CQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: <nil> Jul 26 23:12:45.881283472 (59h36m55.603846512s) [noise=true , address=79.127.201.214:4321 , piece_id=6TIYGXIEDHRT4TCLTEZ4XNJC5XAYAJ3IRSUNJBTPZ2DHH4QNV7SA, seg_pos={0 0}, upload=57194]: Done uploading piece. err: <nil> Jul 26 23:12:51.038964110 (59h37m0.76152714s) [noise=true , address=77.90.31.16:4766 , piece_id=WKSRDTYA54FXRLQE7G4GZPMFFJGR2ZQFHDSIQUBPPFEIQRJZ4YSQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: <nil> Jul 26 23:12:55.893879181 (59h37m5.616442211s) [noise=true , address=77.90.31.15:4721 , piece_id=OAP7NKIKFWFWIRWJWQL5QT5OBCCQX55STOQBAHNYZSUU4DNLXFMQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: <nil> Jul 26 23:13:36.095928483 (59h37m45.818491513s) [noise=true , address=77.90.31.10:4533 , piece_id=FEVPHEKTNEXEECHDBWNA6527KTLNX64IZXIVZXGZ65WWE64G7G4Q, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:13:36.096027523 (59h37m45.818590563s) [noise=true , address=104.36.82.53:4135 , piece_id=RRFL56VSJOR6CH5V7X5B7T37XS3M4RE52ZRNQTCXZT7IQYBY2DYA, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: rpc: dial tcp 104.36.82.53:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:36.096077078 (59h37m45.818640118s) [noise=true , address=104.36.82.15:4266 , piece_id=TVOAVS32BMXXZUO3FQHFYUXJ2MQCIQOIYMJ4AMFT2S7T32O5OSSQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:1wH8jka9Jw8JLYdGKuit8XoJjVLJRDKJ5ePgeei16eax6mk3j, address:104.36.82.15:4266): protocol: rpc: dial tcp 104.36.82.15:4266: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:36.096118677 (59h37m45.818681717s) [noise=true , address=104.36.82.52:4105 , piece_id=R6DZELQO36N3NPB2PETSUD374OTSA3LVCGQTDZM7GYLJE465F7CQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:104.36.82.52:4105): protocol: write tcp 188.166.225.170:52908->104.36.82.52:4105: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:36.096197218 (59h37m45.818803922s) [noise=true , address=104.36.82.46:4494 , piece_id=MAAHQZOM2LDU4C5IBU72LGAGFZMWBYPBSIRS7JSJIX2TGSQBGJ5A, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:53396->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:36.096287030 (59h37m45.81885007s) [noise=true , address=104.36.82.47:4552 , piece_id=BQF777JA3H26YZFZ5MARKY4SXTRPHLT5HY7NRHSIYKUKLOYTDEXA, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:36.096311486 (59h37m45.818874526s) [noise=true , address=104.36.82.54:4168 , piece_id=YHOHRO5S7VLDIW3HW2RVEPE2ANRASG25WKEHT47UPXWO5MOA22CA, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: write tcp 188.166.225.170:55002->104.36.82.54:4168: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:36.096356483 (59h37m45.818919513s) [noise=true , address=104.36.82.55:4197 , piece_id=HDR3KAUPCN7WV462SEOPJMFAU2FLLBE4JQY622ZZWA7Y5J5WNUKA, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: rpc: dial tcp 104.36.82.55:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:36.096463898 (59h37m45.819026928s) [noise=true , address=104.36.82.56:4249 , piece_id=T374HUIJB52XONPQQ33B3N74BUHDPX52RKCMYW4H72JDE6F6AAPQ, seg_pos={0 0}, upload=57194]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 188.166.225.170:58958->104.36.82.56:4249: use of closed network connection; context deadline exceeded Jul 26 23:13:36.096483827 (59h37m45.819046867s) [seg_pos={0 0}, upload=57194]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:13:36.321769777 (59h37m46.044332817s) [upload=57194]: Done uploading segments... Jul 26 23:13:36.321825063 (59h37m46.044388103s) [upload=57194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.227548389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXX.X..X] duration: 30.001896934s FAIL (2025-07-26 23:13:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001918275s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..........X.X........X.X...........................XXXXXXXX] duration: 30.464189098s FAIL (2025-07-26 23:13:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.46420011s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.997141317s FAIL (2025-07-26 23:13:07 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.997165825s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....................X.........X...........X.......XXXXXXXXX] duration: 30.236702921s FAIL (2025-07-26 23:13:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s={0 0}, upload=57182]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:22.333669842 (59h36m32.056232882s) [seg_pos={0 0}, upload=57182]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:12:22.568180432 (59h36m32.290743472s) [upload=57182]: Done uploading segments... Jul 26 23:12:22.568236890 (59h36m32.29079994s) [upload=57182]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:12:34.439521304 (59h36m44.162084344s) [upload=57192]: Starting upload Jul 26 23:12:34.439575107 (59h36m44.162138147s) [upload=57192]: Done starting upload Jul 26 23:12:34.439691300 (59h36m44.16225434s) [upload=57192]: Uploading segments... Jul 26 23:12:34.439792383 (59h36m44.162355423s) [upload=57192]: Got next segment. Inline: false Jul 26 23:12:34.439826067 (59h36m44.162389117s) [upload=57192]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:12:34.439831318 (59h36m44.162394368s) [upload=57192]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:12:34.671340353 (59h36m44.393903383s) [seg_pos={0 0}, upload=57192]: Begin upload segment... Jul 26 23:12:34.671379747 (59h36m44.393942788s) [seg_pos={0 0}, upload=57192]: Done begin upload segment. Jul 26 23:12:34.671403072 (59h36m44.393966112s) [noise=true , address=45.91.116.11:4117 , piece_id=HZMQBHK7B2W2CMAGYHCN7CJH6RB7HPPYNM4RZFK72ZFZU4CCESFQ, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.671430174 (59h36m44.393993214s) [noise=true , address=45.91.116.12:4226 , piece_id=HNPCRAJ7EYJ4XEJC63HKEXB2HVEFOQXWFCJXX6ZV35WEPVKYIXYQ, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.671743864 (59h36m44.394306894s) [noise=true , address=77.90.31.14:4662 , piece_id=KV367QAM26K2KK6L4OM5SRPTBKPT4EJSQXNDNN7GK7PMFX2BEKOA, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.671892038 (59h36m44.394455078s) [noise=true , address=104.36.82.4:4802 , piece_id=ZSRJVR6V3WIFWEDZYMOUDEVHMV7IURPY77BRJVNCSSL42P26K3VA, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.672011767 (59h36m44.394574807s) [noise=true , address=45.91.116.6:4975 , piece_id=W2PO4KULXJGVC36GBIDSRXZX7BUUJA2AIYGPHTYNUP3IZBGDUKSA, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.672378649 (59h36m44.394941689s) [noise=true , address=104.36.82.2:4730 , piece_id=43VR75IMRQX3TTEYD4JLV7UWHOMLR5YPLHVJ4BFCDCCLVLSIYSYQ, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.672397805 (59h36m44.394960845s) [noise=true , address=104.36.82.15:4258 , piece_id=CPHX7DDAWDCNYGDOE7IHTEDCXR5HREEKYAURJVSEZ4PRU6VWQPVQ, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.672660478 (59h36m44.395223518s) [noise=true , address=104.36.82.15:4257 , piece_id=GM6ZV2Y4J42X5XGXLGDVHIFX7MCR3WYZH2XNBSJSOLIWFDVNDWQA, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.672901460 (59h36m44.3954645s) [noise=true , address=45.91.116.10:4083 , piece_id=WKSUUBMIPGSAUAWAYMHNTGWPCEJMTJY32QTH53DLSWWBUKODYCOA, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.673106882 (59h36m44.395669922s) [noise=true , address=104.36.82.6:4874 , piece_id=GLNV554YAAUJUBKJ4NLH7LMYGDLZZWRXDAV7W6EWXDS6TJIPH7VQ, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.673324188 (59h36m44.395887228s) [noise=true , address=77.90.31.14:4661 , piece_id=6TH6WRMG4WPIXNS7T767FNNYNJACCGMJ2NWWQY3NZAPGYX66JPBQ, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.673531806 (59h36m44.396094846s) [noise=true , address=45.91.116.10:4081 , piece_id=LAYNSHLSJQG7TYHQ3CT223QFVB3HTN5LJHA7DDZXJO3PNIMODACQ, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.673739853 (59h36m44.396302893s) [noise=true , address=45.91.116.6:4973 , piece_id=KC3EFWGILFQCSCL33VKDHD3MJVONRNFDWEI3BOFUK3BGSE7LYHFA, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.673954925 (59h36m44.396517965s) [noise=true , address=45.91.116.4:4939 , piece_id=CLIKSV4WVQZPZKLDXDG4H7VE3U5N5WU25IUD737ZYWTON6CLWJ5A, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.674026452 (59h36m44.396589492s) [noise=true , address=45.91.116.4:4940 , piece_id=MHSI4JRO5SXHAQ5PWAIYA4EUVNXL4EDXZ5DMIIYA7MPH3BLIU42Q, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:34.674285117 (59h36m44.396848157s) [noise=true , address=45.91.116.8:4047 , piece_id=CBFWGZ5424BAVKNLVSRQTRLDOYID6EPDAB3CKA4P4Q67OSXGG55Q, seg_pos={0 0}, upload=57192]: Uploading piece... Jul 26 23:12:36.312762313 (59h36m46.035325343s) [upload=57192]: Next returned nil segment Jul 26 23:12:36.312778072 (59h36m46.035341112s) [upload=57192]: Waiting for error group managing segments... Jul 26 23:12:36.701679379 (59h36m46.424242419s) [noise=true , address=45.91.116.12:4226 , piece_id=HNPCRAJ7EYJ4XEJC63HKEXB2HVEFOQXWFCJXX6ZV35WEPVKYIXYQ, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.702959529 (59h36m46.425522569s) [noise=true , address=45.91.116.10:4083 , piece_id=WKSUUBMIPGSAUAWAYMHNTGWPCEJMTJY32QTH53DLSWWBUKODYCOA, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.769671248 (59h36m46.492234279s) [noise=true , address=77.90.31.14:4661 , piece_id=6TH6WRMG4WPIXNS7T767FNNYNJACCGMJ2NWWQY3NZAPGYX66JPBQ, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.774170587 (59h36m46.496733648s) [noise=true , address=45.91.116.4:4940 , piece_id=MHSI4JRO5SXHAQ5PWAIYA4EUVNXL4EDXZ5DMIIYA7MPH3BLIU42Q, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.775073815 (59h36m46.497636855s) [noise=true , address=45.91.116.4:4939 , piece_id=CLIKSV4WVQZPZKLDXDG4H7VE3U5N5WU25IUD737ZYWTON6CLWJ5A, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.780853093 (59h36m46.503416133s) [noise=true , address=45.91.116.8:4047 , piece_id=CBFWGZ5424BAVKNLVSRQTRLDOYID6EPDAB3CKA4P4Q67OSXGG55Q, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.786074302 (59h36m46.508637342s) [noise=true , address=77.90.31.14:4662 , piece_id=KV367QAM26K2KK6L4OM5SRPTBKPT4EJSQXNDNN7GK7PMFX2BEKOA, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.833740021 (59h36m46.556303051s) [noise=true , address=45.91.116.6:4973 , piece_id=KC3EFWGILFQCSCL33VKDHD3MJVONRNFDWEI3BOFUK3BGSE7LYHFA, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.857024653 (59h36m46.579587693s) [noise=true , address=45.91.116.6:4975 , piece_id=W2PO4KULXJGVC36GBIDSRXZX7BUUJA2AIYGPHTYNUP3IZBGDUKSA, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.868932506 (59h36m46.591495546s) [noise=true , address=45.91.116.11:4117 , piece_id=HZMQBHK7B2W2CMAGYHCN7CJH6RB7HPPYNM4RZFK72ZFZU4CCESFQ, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:12:36.922674310 (59h36m46.64523735s) [noise=true , address=45.91.116.10:4081 , piece_id=LAYNSHLSJQG7TYHQ3CT223QFVB3HTN5LJHA7DDZXJO3PNIMODACQ, seg_pos={0 0}, upload=57192]: Done uploading piece. err: <nil> Jul 26 23:13:03.799209999 (59h37m13.521773039s) [noise=true , address=104.36.82.6:4874 , piece_id=GLNV554YAAUJUBKJ4NLH7LMYGDLZZWRXDAV7W6EWXDS6TJIPH7VQ, seg_pos={0 0}, upload=57192]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:50128->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:03.799351430 (59h37m13.52191447s) [noise=true , address=104.36.82.4:4802 , piece_id=ZSRJVR6V3WIFWEDZYMOUDEVHMV7IURPY77BRJVNCSSL42P26K3VA, seg_pos={0 0}, upload=57192]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:03.799413708 (59h37m13.521976748s) [noise=true , address=104.36.82.2:4730 , piece_id=43VR75IMRQX3TTEYD4JLV7UWHOMLR5YPLHVJ4BFCDCCLVLSIYSYQ, seg_pos={0 0}, upload=57192]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:03.799478161 (59h37m13.522041201s) [noise=true , address=104.36.82.15:4258 , piece_id=CPHX7DDAWDCNYGDOE7IHTEDCXR5HREEKYAURJVSEZ4PRU6VWQPVQ, seg_pos={0 0}, upload=57192]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:03.799505774 (59h37m13.522068804s) [noise=true , address=104.36.82.15:4257 , piece_id=GM6ZV2Y4J42X5XGXLGDVHIFX7MCR3WYZH2XNBSJSOLIWFDVNDWQA, seg_pos={0 0}, upload=57192]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:60500->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:03.799526644 (59h37m13.522089684s) [seg_pos={0 0}, upload=57192]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:13:04.033366762 (59h37m13.755929862s) [upload=57192]: Done uploading segments... Jul 26 23:13:04.033396890 (59h37m13.75595994s) [upload=57192]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.236728782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..........X.X........X.X...........................XXXXXXX] duration: 30.230048523s FAIL (2025-07-26 23:12:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.23006748s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.255079735s FAIL (2025-07-26 23:12:44 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.255101556s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...X...............X..XX...X.X.X.X.......XX..XX.XX..XXX.....] duration: 28.970232485s FAIL (2025-07-26 23:12:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.970251751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXX] duration: 1m0.232484854s FAIL (2025-07-26 23:12:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== VFL2ZY2AMNUJRK7JWUF5PA4B3RGK7YHVTQFXHANZUIZZYDQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.697111531 (59h35m46.419674561s) [noise=true , address=104.36.82.52:4119 , piece_id=SULNAWT6SMQIIS4ZRM72UOQPGN33GVKXEHO6PQRXKH3FXECJSCYQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.697352862 (59h35m46.419915892s) [noise=true , address=104.36.82.47:4530 , piece_id=QD3TYXST3KD47H74CBF3QNIP6EUWLHLSUHW62N7LJOVCWQDCGFNA, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.697393110 (59h35m46.41995615s) [noise=true , address=104.36.82.56:4239 , piece_id=CUDF5OR3DR2D3ZBP44K522EF6O243ZAFABWHMBLDOPUPTJWAGSVA, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.697673927 (59h35m46.420236967s) [noise=true , address=64.49.13.140:4514 , piece_id=FGBPOS5J5MZIGFIUGDHVT36NSOGAZGAEYKKPZAXQ3RU72WZ6Q7YQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.697723622 (59h35m46.420286662s) [noise=true , address=77.90.31.10:4525 , piece_id=UC5RNXJ5OTV6UJKKHGU6BG4KBWIC2GMTXPVHPFLMOEY6XOYJ4OZQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.698007415 (59h35m46.420570455s) [noise=true , address=79.127.205.234:4603 , piece_id=4CVTGJ6SEJHJDJU7CBPFXN2XTG2ZIWLFX7FYIPDDRETZMONENDSQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.698026131 (59h35m46.420589171s) [noise=true , address=104.36.82.55:4198 , piece_id=AHLE7UMPZCUSOTO4WPAHQXR3PID2NDWFUPILI3QRUYZV4DSLASYA, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.698325233 (59h35m46.420888273s) [noise=true , address=79.127.219.48:4090 , piece_id=7WJFWGUTCPZEP3QVEGFHFI6IYUE2HOFCC5ACY6DKZWC5MZG4RFBQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.698556305 (59h35m46.421119365s) [noise=true , address=104.36.82.23:4451 , piece_id=7YTFTFSLYKKPESJ6V6QV5TOXUQJ5FCPHMPU6IYVJQPRHRV7PX4FQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.698636849 (59h35m46.421199879s) [noise=true , address=64.49.13.139:4478 , piece_id=66VBUYWXWYJL5UJN5HOTU3A2OZGS4RLQBOWF7TTNGBB363SZ3XNQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.698892778 (59h35m46.421455819s) [noise=true , address=104.36.82.60:4363 , piece_id=GDO7GG5LPKF2FNS6XOFGZK42TJZTBM73SNLCN2MLGGEW73JNE3QA, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.699129923 (59h35m46.421692963s) [noise=true , address=104.36.82.53:4148 , piece_id=HUGCEMVIBSLB4DU4GSPZX6W6A5CELITAU6A52RLCY6QYGJWI2H6Q, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.699171643 (59h35m46.421734683s) [noise=true , address=79.127.205.233:4643 , piece_id=I2CQ52WYMDCLC7O27AAMOYFOXTK3BFZDKST5JO3XARH44C2EHBYQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.699434205 (59h35m46.421997245s) [noise=true , address=77.90.31.21:4908 , piece_id=7VI2RK5EM6QBVOM6OGCD5USJBGHAIUGSEZUNI3HL5GF4LZE7L36A, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.699453873 (59h35m46.422016913s) [noise=true , address=104.36.82.54:4167 , piece_id=5WSK5XQCUAGESZWOKHBY4L4T5UNM5KGIQ6A5X5MGSIXLWJ6WRWKA, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:36.699677330 (59h35m46.42224037s) [noise=true , address=79.127.219.51:4210 , piece_id=IQ5QDJBIQ7PT4QJ2URJYD2AZXAHHGUJWO4RGLOTLGVJHCO2WMSXQ, seg_pos={0 0}, upload=57177]: Uploading piece... Jul 26 23:11:37.798533235 (59h35m47.521096275s) [noise=true , address=104.36.82.60:4363 , piece_id=GDO7GG5LPKF2FNS6XOFGZK42TJZTBM73SNLCN2MLGGEW73JNE3QA, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: storage node overloaded, request limit: 2000 Jul 26 23:11:37.822551760 (59h35m47.5451148s) [upload=57177]: Next returned nil segment Jul 26 23:11:37.822569695 (59h35m47.545132735s) [upload=57177]: Waiting for error group managing segments... Jul 26 23:11:38.108172810 (59h35m47.83073587s) [noise=true , address=79.127.205.234:4603 , piece_id=4CVTGJ6SEJHJDJU7CBPFXN2XTG2ZIWLFX7FYIPDDRETZMONENDSQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:11:38.117019577 (59h35m47.839582617s) [noise=true , address=79.127.205.233:4643 , piece_id=I2CQ52WYMDCLC7O27AAMOYFOXTK3BFZDKST5JO3XARH44C2EHBYQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:11:38.135116933 (59h35m47.857679963s) [noise=true , address=64.49.13.139:4478 , piece_id=66VBUYWXWYJL5UJN5HOTU3A2OZGS4RLQBOWF7TTNGBB363SZ3XNQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:11:38.673731827 (59h35m48.396294887s) [noise=true , address=77.90.31.21:4908 , piece_id=7VI2RK5EM6QBVOM6OGCD5USJBGHAIUGSEZUNI3HL5GF4LZE7L36A, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:11:40.639670094 (59h35m50.362233134s) [noise=true , address=79.127.219.51:4210 , piece_id=IQ5QDJBIQ7PT4QJ2URJYD2AZXAHHGUJWO4RGLOTLGVJHCO2WMSXQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:11:42.379798721 (59h35m52.102361761s) [noise=true , address=104.36.82.23:4451 , piece_id=7YTFTFSLYKKPESJ6V6QV5TOXUQJ5FCPHMPU6IYVJQPRHRV7PX4FQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:11:42.459064519 (59h35m52.181627549s) [noise=true , address=79.127.219.48:4090 , piece_id=7WJFWGUTCPZEP3QVEGFHFI6IYUE2HOFCC5ACY6DKZWC5MZG4RFBQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:11:42.996759694 (59h35m52.719322724s) [noise=true , address=64.49.13.140:4514 , piece_id=FGBPOS5J5MZIGFIUGDHVT36NSOGAZGAEYKKPZAXQ3RU72WZ6Q7YQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:12:04.035817582 (59h36m13.758380622s) [noise=true , address=77.90.31.10:4525 , piece_id=UC5RNXJ5OTV6UJKKHGU6BG4KBWIC2GMTXPVHPFLMOEY6XOYJ4OZQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: <nil> Jul 26 23:12:35.607215839 (59h36m45.329778879s) [noise=true , address=104.36.82.52:4119 , piece_id=SULNAWT6SMQIIS4ZRM72UOQPGN33GVKXEHO6PQRXKH3FXECJSCYQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:12mTLjL6Y6VVjqztwVXYif4v7tyftjLafztLUibajksMHCFraFG, address:104.36.82.52:4119): protocol: rpc: dial tcp 104.36.82.52:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:35.607243842 (59h36m45.329806882s) [noise=true , address=104.36.82.46:4509 , piece_id=RAFQ3VFL2ZY2AMNUJRK7JWUF5PA4B3RGK7YHVTQFXHANZUIZZYDQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: write tcp 188.166.225.170:34686->104.36.82.46:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:35.607276184 (59h36m45.329839224s) [noise=true , address=104.36.82.53:4148 , piece_id=HUGCEMVIBSLB4DU4GSPZX6W6A5CELITAU6A52RLCY6QYGJWI2H6Q, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:35.607304528 (59h36m45.329867588s) [noise=true , address=104.36.82.56:4239 , piece_id=CUDF5OR3DR2D3ZBP44K522EF6O243ZAFABWHMBLDOPUPTJWAGSVA, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: rpc: dial tcp 104.36.82.56:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:35.607379943 (59h36m45.329942983s) [noise=true , address=104.36.82.55:4198 , piece_id=AHLE7UMPZCUSOTO4WPAHQXR3PID2NDWFUPILI3QRUYZV4DSLASYA, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: rpc: dial tcp 104.36.82.55:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:35.607412344 (59h36m45.329975384s) [noise=true , address=77.90.31.17:4800 , piece_id=6ZPTYOYZP645KZ3H7QKJAPRCPIGLDKJ7DE2LD5RSN6JNU32VN4EQ, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: read tcp 188.166.225.170:46608->77.90.31.17:4800: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:35.607423416 (59h36m45.329986456s) [noise=true , address=104.36.82.47:4530 , piece_id=QD3TYXST3KD47H74CBF3QNIP6EUWLHLSUHW62N7LJOVCWQDCGFNA, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: write tcp 188.166.225.170:36776->104.36.82.47:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:35.607427494 (59h36m45.329990534s) [noise=true , address=104.36.82.54:4167 , piece_id=5WSK5XQCUAGESZWOKHBY4L4T5UNM5KGIQ6A5X5MGSIXLWJ6WRWKA, seg_pos={0 0}, upload=57177]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: write tcp 188.166.225.170:37218->104.36.82.54:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:35.607455938 (59h36m45.330018968s) [seg_pos={0 0}, upload=57177]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:12:35.837602078 (59h36m45.560165118s) [upload=57177]: Done uploading segments... Jul 26 23:12:35.837626875 (59h36m45.560189915s) [upload=57177]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.232511184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......................X.........X...........X.......XXXXXXXX] duration: 30.236185934s FAIL (2025-07-26 23:12:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== al: 13 errs: <nil> Jul 26 23:11:51.239414501 (59h36m0.961977541s) [upload=57174]: Done uploading segments... Jul 26 23:11:51.239439549 (59h36m0.962002589s) [upload=57174]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:11:52.973972047 (59h36m2.696535087s) [upload=57182]: Starting upload Jul 26 23:11:52.974058012 (59h36m2.696621052s) [upload=57182]: Done starting upload Jul 26 23:11:52.974165948 (59h36m2.696728978s) [upload=57182]: Uploading segments... Jul 26 23:11:52.974265248 (59h36m2.696828278s) [upload=57182]: Got next segment. Inline: false Jul 26 23:11:52.974306898 (59h36m2.696869938s) [upload=57182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:11:52.974311767 (59h36m2.696874807s) [upload=57182]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:11:53.209129875 (59h36m2.931692915s) [seg_pos={0 0}, upload=57182]: Begin upload segment... Jul 26 23:11:53.209173188 (59h36m2.931736228s) [seg_pos={0 0}, upload=57182]: Done begin upload segment. Jul 26 23:11:53.209195090 (59h36m2.93175812s) [noise=true , address=104.36.82.4:4801 , piece_id=RFAOSSERLHJT23KLZ3XZE5MY4UGKQYB2HGIXXPB3M6KFSY5ITCGQ, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.209240557 (59h36m2.931803587s) [noise=true , address=45.91.116.6:4975 , piece_id=DZUYGLNBPYSUJGQPOISARFO6274FQBLIQO32VLSPQLCXY4NLG7JA, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.209563365 (59h36m2.932126405s) [noise=true , address=45.91.116.7:4011 , piece_id=JUE7RBR3CSOT2TYAMZ7BRMDEV6P2HMHZRL5J6NK66CCA35DMQPHA, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.209732068 (59h36m2.932295108s) [noise=true , address=104.36.82.2:4731 , piece_id=5CBDVUTQ5X5GLW3JZB7Z2JZVDMAP3VFPNAZ46JCBEQBTCINTD2EQ, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.209886553 (59h36m2.932449603s) [noise=true , address=45.91.116.7:4009 , piece_id=7DKKFSBQ6QMITBHLCITWQR7YI3SSAFYF6ZWJRKBZDRSBUV2OHQKQ, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.210044996 (59h36m2.932608036s) [noise=true , address=45.91.116.8:4045 , piece_id=SWCFB3VAD5OAEEJGD4QRH4VBDZCEF7VUNDGT2EZOV4BEUBYIC42A, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.210162441 (59h36m2.932725482s) [noise=true , address=104.36.82.6:4874 , piece_id=O7XBEEY7IMSUWJAD5CJBVF7VID4CKNDJ4ACCIFZ3FHY2GNWLS4AQ, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.210324471 (59h36m2.932887511s) [noise=true , address=45.91.116.12:4225 , piece_id=OVEBZDBS2XCV37JEXP35PATMZWAC75TAPEUUNAQJUH55HHRZUX7A, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.210405155 (59h36m2.932968195s) [noise=true , address=104.36.82.4:4802 , piece_id=ZBR7O6OBHQJ3STEFSWEMONLNRU355KVF7LWWR7P23TLC5OWBA62A, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.210628632 (59h36m2.933191673s) [noise=true , address=45.91.116.4:4939 , piece_id=XOO3WPJ7YBA66AWDKEGJ656R5VU6NGNU5LYQ3FUIVFCJ6W46MHDA, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.210646547 (59h36m2.933209587s) [noise=true , address=77.90.31.14:4661 , piece_id=J4IIJ5I7IYHVFGG4AOQ5333D7DA5RDPP34HVOKCKM2UF26LIUJRA, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.210899701 (59h36m2.933462741s) [noise=true , address=104.36.82.3:4765 , piece_id=3AZMWVYTFXKZP6IXJ4P6RRYXN4XTF5C55UJIKKOPBFM6X7EJTYDQ, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.210966169 (59h36m2.933529209s) [noise=true , address=104.36.82.3:4767 , piece_id=VGIOMRMNVWBCW5C36J4HW366EONAH3QB32BY2KSUZ4RZNM6LJXSQ, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.211108842 (59h36m2.933671882s) [noise=true , address=104.36.82.15:4257 , piece_id=XQ4MJGD7GFQTAMXCYXRS72W6WXH5GSYEIMJ4RKXPJAIWACH56KFQ, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.211213633 (59h36m2.933776673s) [noise=true , address=104.36.82.16:4295 , piece_id=3TGHFZT43OVRUROQPHBTKQPJK24JROU5ZJ66PKZNGIWCFOEL4K7Q, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:53.211358159 (59h36m2.933921189s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGJ7MJ5TBTQ2GWZJNVBF6GI6RQUKFA3XFKYKWF43LG5O3CMMCT2Q, seg_pos={0 0}, upload=57182]: Uploading piece... Jul 26 23:11:54.980857358 (59h36m4.703420398s) [upload=57182]: Next returned nil segment Jul 26 23:11:54.980876394 (59h36m4.703439434s) [upload=57182]: Waiting for error group managing segments... Jul 26 23:11:55.258697180 (59h36m4.981260221s) [noise=true , address=45.91.116.8:4045 , piece_id=SWCFB3VAD5OAEEJGD4QRH4VBDZCEF7VUNDGT2EZOV4BEUBYIC42A, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:11:55.314407070 (59h36m5.03697011s) [noise=true , address=77.90.31.14:4661 , piece_id=J4IIJ5I7IYHVFGG4AOQ5333D7DA5RDPP34HVOKCKM2UF26LIUJRA, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:11:55.325406146 (59h36m5.047969176s) [noise=true , address=45.91.116.7:4011 , piece_id=JUE7RBR3CSOT2TYAMZ7BRMDEV6P2HMHZRL5J6NK66CCA35DMQPHA, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:11:55.326490881 (59h36m5.049053921s) [noise=true , address=45.91.116.7:4009 , piece_id=7DKKFSBQ6QMITBHLCITWQR7YI3SSAFYF6ZWJRKBZDRSBUV2OHQKQ, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:11:55.335448820 (59h36m5.05801186s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGJ7MJ5TBTQ2GWZJNVBF6GI6RQUKFA3XFKYKWF43LG5O3CMMCT2Q, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:11:55.342190198 (59h36m5.064753238s) [noise=true , address=45.91.116.4:4939 , piece_id=XOO3WPJ7YBA66AWDKEGJ656R5VU6NGNU5LYQ3FUIVFCJ6W46MHDA, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:11:55.371686112 (59h36m5.094249152s) [noise=true , address=45.91.116.6:4975 , piece_id=DZUYGLNBPYSUJGQPOISARFO6274FQBLIQO32VLSPQLCXY4NLG7JA, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:11:55.379728049 (59h36m5.102291079s) [noise=true , address=45.91.116.12:4225 , piece_id=OVEBZDBS2XCV37JEXP35PATMZWAC75TAPEUUNAQJUH55HHRZUX7A, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:11:55.426621130 (59h36m5.14918417s) [noise=true , address=104.36.82.16:4295 , piece_id=3TGHFZT43OVRUROQPHBTKQPJK24JROU5ZJ66PKZNGIWCFOEL4K7Q, seg_pos={0 0}, upload=57182]: Done uploading piece. err: <nil> Jul 26 23:12:22.333476092 (59h36m32.056039132s) [noise=true , address=104.36.82.3:4765 , piece_id=3AZMWVYTFXKZP6IXJ4P6RRYXN4XTF5C55UJIKKOPBFM6X7EJTYDQ, seg_pos={0 0}, upload=57182]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:22.333512883 (59h36m32.056075923s) [noise=true , address=104.36.82.4:4801 , piece_id=RFAOSSERLHJT23KLZ3XZE5MY4UGKQYB2HGIXXPB3M6KFSY5ITCGQ, seg_pos={0 0}, upload=57182]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:22.333552438 (59h36m32.056115478s) [noise=true , address=104.36.82.2:4731 , piece_id=5CBDVUTQ5X5GLW3JZB7Z2JZVDMAP3VFPNAZ46JCBEQBTCINTD2EQ, seg_pos={0 0}, upload=57182]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:53736->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:22.333585641 (59h36m32.056148681s) [noise=true , address=104.36.82.3:4767 , piece_id=VGIOMRMNVWBCW5C36J4HW366EONAH3QB32BY2KSUZ4RZNM6LJXSQ, seg_pos={0 0}, upload=57182]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 188.166.225.170:37728->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:22.333591452 (59h36m32.056154492s) [noise=true , address=104.36.82.15:4257 , piece_id=XQ4MJGD7GFQTAMXCYXRS72W6WXH5GSYEIMJ4RKXPJAIWACH56KFQ, seg_pos={0 0}, upload=57182]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:22.333614416 (59h36m32.056177446s) [noise=true , address=104.36.82.6:4874 , piece_id=O7XBEEY7IMSUWJAD5CJBVF7VID4CKNDJ4ACCIFZ3FHY2GNWLS4AQ, seg_pos={0 0}, upload=57182]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:22.333645135 (59h36m32.056208175s) [noise=true , address=104.36.82.4:4802 , piece_id=ZBR7O6OBHQJ3STEFSWEMONLNRU355KVF7LWWR7P23TLC5OWBA62A, seg_pos={0 0}, upload=57182]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:22.333669842 (59h36m32.056232882s) [seg_pos={0 0}, upload=57182]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:12:22.568180432 (59h36m32.290743472s) [upload=57182]: Done uploading segments... Jul 26 23:12:22.568236890 (59h36m32.29079994s) [upload=57182]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.236201874s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXX.X..] duration: 15.808127568s FAIL (2025-07-26 23:12:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.808148269s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)