Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23181715s FAIL (2025-07-27 00:12:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== piece... Jul 27 00:11:31.017033996 (60h35m40.739597036s) [noise=true , address=79.127.205.245:4240 , piece_id=WJ255QQBQWPBQS5DMJ72MJ6IM2JVTVFD6WMKHYT7HR46NZP6EO3Q, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017246372 (60h35m40.739809412s) [noise=true , address=77.90.31.18:4817 , piece_id=4ZXCWQQ2WLGENBRCFI3B7IIT5AWH6X6IIJ3KVAHKLOWO2CAJRDZQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017307839 (60h35m40.739870879s) [noise=true , address=79.127.205.234:4604 , piece_id=T2HKHTZDWBVRGMMU5WMSNXURMHQWKSOXWJW5BH75RULVWTIIDCSA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017570702 (60h35m40.740133733s) [noise=true , address=104.36.82.15:4284 , piece_id=CKPI3HIJO6IOLQAQXM3MJSL3KIOI5Z5WOZMWDEDGIKA2GJUQCDNQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017637410 (60h35m40.74020045s) [noise=true , address=64.49.13.26:4953 , piece_id=OF23F7MJ4V3RNCCWIEBC2QPNYFD4VAYX6MDMRI4MBD3QENNPB5OA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017870216 (60h35m40.740433256s) [noise=true , address=79.127.220.101:4041 , piece_id=AXPE5CYLHBMO5YVDR2R6EZXE4FO73EOUDYTAAH7QRR5L5XCD6QEQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017953896 (60h35m40.740516936s) [noise=true , address=77.90.31.13:4658 , piece_id=TNKDY7OTTOSQVHSSV7MIKRWK4WSL3OWVTIL7UICXX4OPMDHJA6IA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018152706 (60h35m40.740715746s) [noise=true , address=64.49.13.140:4508 , piece_id=253UHTR2WJEXZLBBUVZSZPXTNB5BCJCJESX5GNJM3U5LAJYJZ7YA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018210096 (60h35m40.740773136s) [noise=true , address=79.127.205.235:4685 , piece_id=SQF2VPXAXGKKCR5A672XHF7LG3TA6T6OZ7BGJG563OZYF5A5AXZQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018371675 (60h35m40.740934705s) [noise=true , address=104.36.82.46:4512 , piece_id=MYVSMBZ3MKSFUZ3R6PYLTPMGOMBFQ5J5L2MCTDUTMSHER4VX4GTQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018440778 (60h35m40.741003818s) [noise=true , address=77.90.31.16:4748 , piece_id=QI4KIP6JGETWMIFSIOWA366AYQ72MVTG44AP24CC5WPN76IVPCQQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018729870 (60h35m40.74129291s) [noise=true , address=104.36.82.51:4062 , piece_id=2V3OXLJEHQN424CALZ34TSBMGHYVCTS7ESDADE53YMZYLFBJP7IQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018981191 (60h35m40.741544221s) [noise=true , address=79.127.205.246:4274 , piece_id=BLXPNB57JPUTAXHLBOG3NCDP7VWSP25CV3LRGQJOG2LXCSUPBW6Q, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019211282 (60h35m40.741774322s) [noise=true , address=104.36.82.12:4168 , piece_id=CDFBMTTRNE7JCWHNTHBM2QS4S2R6NOLFL3HHWECJJJ43OJXAJZXQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019456681 (60h35m40.742019721s) [noise=true , address=104.36.82.20:4461 , piece_id=LEQQH3P2GD6K3ILMZRSZOCEIUFG6MU5R25X4MHZORL3URWIMSWMQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019673756 (60h35m40.742236796s) [noise=true , address=79.127.219.51:4210 , piece_id=5KLO3A4G555YCTZ7QCZVX3JWCMP4EESBUVUDU24LZFCKWZTB2EYA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019901743 (60h35m40.742464783s) [noise=true , address=104.36.82.47:4537 , piece_id=WJNHUEACCBGUZQ4MHHKCSK7AEDX34G5Z5K2FRZHVAFKWNJFG3JXA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:32.116889070 (60h35m41.83945211s) [upload=58114]: Next returned nil segment Jul 27 00:11:32.116907094 (60h35m41.839470135s) [upload=58114]: Waiting for error group managing segments... Jul 27 00:11:32.400438007 (60h35m42.123001047s) [noise=true , address=79.127.205.245:4240 , piece_id=WJ255QQBQWPBQS5DMJ72MJ6IM2JVTVFD6WMKHYT7HR46NZP6EO3Q, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.419754955 (60h35m42.142317995s) [noise=true , address=79.127.205.235:4685 , piece_id=SQF2VPXAXGKKCR5A672XHF7LG3TA6T6OZ7BGJG563OZYF5A5AXZQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.454796929 (60h35m42.17735997s) [noise=true , address=79.127.205.246:4274 , piece_id=BLXPNB57JPUTAXHLBOG3NCDP7VWSP25CV3LRGQJOG2LXCSUPBW6Q, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.508361473 (60h35m42.230924514s) [noise=true , address=79.127.205.234:4604 , piece_id=T2HKHTZDWBVRGMMU5WMSNXURMHQWKSOXWJW5BH75RULVWTIIDCSA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:33.092872603 (60h35m42.815435643s) [noise=true , address=77.90.31.16:4748 , piece_id=QI4KIP6JGETWMIFSIOWA366AYQ72MVTG44AP24CC5WPN76IVPCQQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:34.345847555 (60h35m44.068410595s) [noise=true , address=79.127.219.51:4210 , piece_id=5KLO3A4G555YCTZ7QCZVX3JWCMP4EESBUVUDU24LZFCKWZTB2EYA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:36.620976689 (60h35m46.343539729s) [noise=true , address=79.127.220.101:4041 , piece_id=AXPE5CYLHBMO5YVDR2R6EZXE4FO73EOUDYTAAH7QRR5L5XCD6QEQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:36.896156473 (60h35m46.618719513s) [noise=true , address=77.90.31.13:4658 , piece_id=TNKDY7OTTOSQVHSSV7MIKRWK4WSL3OWVTIL7UICXX4OPMDHJA6IA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:40.115868474 (60h35m49.838431514s) [noise=true , address=64.49.13.26:4953 , piece_id=OF23F7MJ4V3RNCCWIEBC2QPNYFD4VAYX6MDMRI4MBD3QENNPB5OA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:49.826606684 (60h35m59.549169724s) [noise=true , address=64.49.13.140:4508 , piece_id=253UHTR2WJEXZLBBUVZSZPXTNB5BCJCJESX5GNJM3U5LAJYJZ7YA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:12:29.931183938 (60h36m39.653746978s) [noise=true , address=104.36.82.46:4512 , piece_id=MYVSMBZ3MKSFUZ3R6PYLTPMGOMBFQ5J5L2MCTDUTMSHER4VX4GTQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: write tcp 188.166.225.170:40114->104.36.82.46:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931219377 (60h36m39.653782487s) [noise=true , address=104.36.82.51:4062 , piece_id=2V3OXLJEHQN424CALZ34TSBMGHYVCTS7ESDADE53YMZYLFBJP7IQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: write tcp 188.166.225.170:48958->104.36.82.51:4062: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931255435 (60h36m39.653818476s) [noise=true , address=104.36.82.54:4173 , piece_id=DN3POBT5YQ5G27ZN2TTHITQD47ZH5M5CK5X5CRWIVP5Y7QKOS2HA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: write tcp 188.166.225.170:55092->104.36.82.54:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931289260 (60h36m39.6538523s) [noise=true , address=77.90.31.18:4817 , piece_id=4ZXCWQQ2WLGENBRCFI3B7IIT5AWH6X6IIJ3KVAHKLOWO2CAJRDZQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 188.166.225.170:33636->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931319177 (60h36m39.653882207s) [noise=true , address=104.36.82.15:4284 , piece_id=CKPI3HIJO6IOLQAQXM3MJSL3KIOI5Z5WOZMWDEDGIKA2GJUQCDNQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1YAnLu8onh19Mx524VikQzCHEFW4U8dgtuLx8amtQAiReSCisF, address:104.36.82.15:4284): protocol: write tcp 188.166.225.170:38452->104.36.82.15:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931322904 (60h36m39.653886004s) [noise=true , address=104.36.82.12:4168 , piece_id=CDFBMTTRNE7JCWHNTHBM2QS4S2R6NOLFL3HHWECJJJ43OJXAJZXQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12XJ5kTKY8pLq3uDJZfFVkTgS56Xx8pG2R5i2A9VRNDEckdKnTF, address:104.36.82.12:4168): protocol: rpc: dial tcp 104.36.82.12:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931339937 (60h36m39.653903017s) [noise=true , address=104.36.82.47:4537 , piece_id=WJNHUEACCBGUZQ4MHHKCSK7AEDX34G5Z5K2FRZHVAFKWNJFG3JXA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931349455 (60h36m39.653912485s) [noise=true , address=104.36.82.20:4461 , piece_id=LEQQH3P2GD6K3ILMZRSZOCEIUFG6MU5R25X4MHZORL3URWIMSWMQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:4461): protocol: rpc: dial tcp 104.36.82.20:4461: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931371597 (60h36m39.653934648s) [seg_pos={0 0}, upload=58114]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:12:30.160452502 (60h36m39.883015582s) [upload=58114]: Done uploading segments... Jul 27 00:12:30.160484543 (60h36m39.883047583s) [upload=58114]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.231844842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX..] duration: 6.231432698s FAIL (2025-07-27 00:12:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.231454349s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.] duration: 14.163498248s FAIL (2025-07-27 00:11:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.163549446s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..] duration: 7.096051247s FAIL (2025-07-27 00:11:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== }, upload=58077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: rpc: dial tcp 104.36.82.3:4765: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.749291092 (60h33m8.471854132s) [seg_pos={0 0}, upload=58077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:08:58.749388238 (60h33m8.471951278s) [upload=58077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:08:58.749395502 (60h33m8.471958542s) [upload=58077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:08:59.071572826 (60h33m8.794135866s) [upload=58077]: Done uploading segments... Jul 27 00:08:59.071590289 (60h33m8.794153319s) [upload=58077]: Upload finished. err: <nil> Jul 27 00:09:23.191431943 (60h33m32.913994983s) [upload=58085]: Starting upload Jul 27 00:09:23.191484463 (60h33m32.914047503s) [upload=58085]: Done starting upload Jul 27 00:09:23.191558114 (60h33m32.914121154s) [upload=58085]: Uploading segments... Jul 27 00:09:23.191660350 (60h33m32.91422345s) [upload=58085]: Got next segment. Inline: false Jul 27 00:09:23.191695326 (60h33m32.914258356s) [upload=58085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:09:23.191699464 (60h33m32.914262504s) [upload=58085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:23.424088299 (60h33m33.146651339s) [seg_pos={0 0}, upload=58085]: Begin upload segment... Jul 27 00:09:23.424132493 (60h33m33.146695533s) [seg_pos={0 0}, upload=58085]: Done begin upload segment. Jul 27 00:09:23.424154705 (60h33m33.146717746s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424193240 (60h33m33.14675628s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424508462 (60h33m33.147071503s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424731339 (60h33m33.147294389s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424777367 (60h33m33.147340397s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425039278 (60h33m33.147602318s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425091207 (60h33m33.147654247s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425286531 (60h33m33.147849571s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425540938 (60h33m33.148103978s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425552049 (60h33m33.148115089s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425763755 (60h33m33.148326785s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425836293 (60h33m33.148399333s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426064771 (60h33m33.148627811s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426081131 (60h33m33.148644171s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426305661 (60h33m33.148868701s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426336901 (60h33m33.148899941s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:25.166110004 (60h33m34.888673044s) [upload=58085]: Next returned nil segment Jul 27 00:09:25.166138388 (60h33m34.888701428s) [upload=58085]: Waiting for error group managing segments... Jul 27 00:09:25.453532572 (60h33m35.176095613s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.457229505 (60h33m35.179792545s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.526191879 (60h33m35.248754919s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.527573101 (60h33m35.250136142s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.531417748 (60h33m35.253980788s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.542134953 (60h33m35.264697993s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.550081446 (60h33m35.272644486s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.608111033 (60h33m35.330674073s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.614659251 (60h33m35.337222291s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.731328030 (60h33m35.453891081s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.884837420 (60h33m36.60740046s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.902326182 (60h33m36.624889222s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:52.543670387 (60h34m2.266233467s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41546->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543696337 (60h34m2.266259427s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543714811 (60h34m2.266277881s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41130->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543776821 (60h34m2.266339921s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:49080->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543813270 (60h34m2.26637632s) [seg_pos={0 0}, upload=58085]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:52.770059330 (60h34m2.49262237s) [upload=58085]: Done uploading segments... Jul 27 00:09:52.770089268 (60h34m2.492652308s) [upload=58085]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.096076656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...] duration: 7.473576427s FAIL (2025-07-27 00:11:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.473602526s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX] duration: 30.000885718s FAIL (2025-07-27 00:11:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00090808s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.953156002s FAIL (2025-07-27 00:11:40 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.953188435s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230953081s FAIL (2025-07-27 00:11:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== piece_id=ENGGRHWRI3OMM4BOAOK7JGHXFPHAG34DO6FQ75I2BIFMAAJ2HL3Q, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.960984242 (60h34m39.683547312s) [noise=true , address=77.90.31.16:4747 , piece_id=XKGI6ZN7EGFFQVDN6R4C65VYA2XI45Q5FES2WG542YIR2KLAL3YA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961282062 (60h34m39.683845102s) [noise=true , address=77.90.31.11:4581 , piece_id=MEVZS3JHH2HI46Y2AZ4RLKJK3I5QD432KBUYI6DFUX632X7XLUPA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961544423 (60h34m39.684107464s) [noise=true , address=104.36.82.52:4115 , piece_id=S6KY6ZLJYXIPQ2YQ4P7EMDGCYJZXNKMDNEGM2YZTWY75HJ2ZB4NQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961578839 (60h34m39.684141879s) [noise=true , address=104.36.82.46:4488 , piece_id=UTRSLBGHKRDZY2LX3OSRCSAV663ZMJ3WZT2V3SIPBHY3D2WG7FKQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961829630 (60h34m39.68439267s) [noise=true , address=79.127.205.246:4278 , piece_id=7LSR475EINMZ2LGKN6DMZKPRCSW2LSEDOAWPYUMRWOXCHIIQCL7A, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961851902 (60h34m39.684414932s) [noise=true , address=77.90.31.18:4808 , piece_id=POFPOA5RP3BZPLCEZFQFGCN54DVAVFJUY4BF7HVIJ6LZHI7OTISQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962092733 (60h34m39.684655773s) [noise=true , address=45.91.116.8:4062 , piece_id=PCM46AIXPQRYH3IFIGCAMAGDVBXRQVAIISVHXFARWRVRQ4QSZPUQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962165031 (60h34m39.684728081s) [noise=true , address=79.127.220.97:4361 , piece_id=2ZLXHTXDVRB4FZ5UAJJIGHAYYLOS3AUCYKBJHLF2ZZX7QVVDIRRQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962360525 (60h34m39.684923565s) [noise=true , address=104.36.82.47:4533 , piece_id=PMYE3EDT3DHWUD67JYXTVW4EGQ5KB7NOWQJ2YAKNIBO5X42WTPRQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962406674 (60h34m39.684969704s) [noise=true , address=104.36.82.38:4689 , piece_id=EB6WFS7RTJAYPH2OC4BOBVD3JXLVTBZTTK3YDWB77YJP6PEZJFHA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962613269 (60h34m39.685176299s) [noise=true , address=45.91.116.9:4277 , piece_id=LNGQZ2L7URAJDOGWPQNF554JJ3GAL53UMHKRZQVOW4YQPMCKLYUA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962675848 (60h34m39.685238888s) [noise=true , address=79.127.205.245:4244 , piece_id=HU7Z5WKSQSKUD5ACVBDX5DXAZJD6LGLNOI33ZC6DW7D7D72U5DLA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962830334 (60h34m39.685393374s) [noise=true , address=79.127.220.98:4422 , piece_id=624SSD7F2OZP5UTM26LTX3O2H2URNRMSEQN6Z2II7XQDVSSIS2DA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962928943 (60h34m39.685491983s) [noise=true , address=77.90.31.19:4846 , piece_id=JBGDSLLASW2MIIRRUXCW7PYK24TWBRLLEBDCNMNAPRMNSJ7RD2WA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.963106011 (60h34m39.685669052s) [noise=true , address=104.36.82.44:4870 , piece_id=IPTBCOOX6CY2LVIBJQAXZGZLQREVQZOEZRJHAQUHHBGNUFVDQ4FQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.963146589 (60h34m39.685709629s) [noise=true , address=79.127.219.47:4048 , piece_id=3AIR5DQG3EWOTVCP4OOXOBMGG2OWF4KDYZORUVB6DIVROL4MD6HA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.965030644 (60h34m39.687593684s) [noise=true , address=79.127.205.233:4640 , piece_id=TBAPYHJTOSXFY4R5CMZNHVP5VIBFYUQ7DQZYUZCE34ZKX66H26ZA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:31.057425385 (60h34m40.779988425s) [upload=58101]: Next returned nil segment Jul 27 00:10:31.057442587 (60h34m40.780005617s) [upload=58101]: Waiting for error group managing segments... Jul 27 00:10:31.358062366 (60h34m41.080625406s) [noise=true , address=79.127.205.246:4278 , piece_id=7LSR475EINMZ2LGKN6DMZKPRCSW2LSEDOAWPYUMRWOXCHIIQCL7A, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.365146409 (60h34m41.087709439s) [noise=true , address=79.127.205.245:4244 , piece_id=HU7Z5WKSQSKUD5ACVBDX5DXAZJD6LGLNOI33ZC6DW7D7D72U5DLA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.394481485 (60h34m41.117044525s) [noise=true , address=79.127.205.233:4640 , piece_id=TBAPYHJTOSXFY4R5CMZNHVP5VIBFYUQ7DQZYUZCE34ZKX66H26ZA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.451086155 (60h34m41.173649195s) [noise=true , address=45.91.116.8:4062 , piece_id=PCM46AIXPQRYH3IFIGCAMAGDVBXRQVAIISVHXFARWRVRQ4QSZPUQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1ZgtJJK3pt2NqSfgFnXjZiYhDafhRdh6WkEf12AFTTuiku3Uhc, address:45.91.116.8:4062): protocol: storage node overloaded, request limit: 2000 Jul 27 00:10:31.944066471 (60h34m41.666629501s) [noise=true , address=77.90.31.16:4747 , piece_id=XKGI6ZN7EGFFQVDN6R4C65VYA2XI45Q5FES2WG542YIR2KLAL3YA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:32.067400052 (60h34m41.789963092s) [noise=true , address=77.90.31.11:4581 , piece_id=MEVZS3JHH2HI46Y2AZ4RLKJK3I5QD432KBUYI6DFUX632X7XLUPA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:34.098541683 (60h34m43.821104723s) [noise=true , address=79.127.220.97:4361 , piece_id=2ZLXHTXDVRB4FZ5UAJJIGHAYYLOS3AUCYKBJHLF2ZZX7QVVDIRRQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.030914207 (60h34m44.753477247s) [noise=true , address=79.127.220.98:4422 , piece_id=624SSD7F2OZP5UTM26LTX3O2H2URNRMSEQN6Z2II7XQDVSSIS2DA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.422624445 (60h34m45.145187485s) [noise=true , address=79.127.219.47:4048 , piece_id=3AIR5DQG3EWOTVCP4OOXOBMGG2OWF4KDYZORUVB6DIVROL4MD6HA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.728360292 (60h34m45.450923322s) [noise=true , address=77.90.31.19:4846 , piece_id=JBGDSLLASW2MIIRRUXCW7PYK24TWBRLLEBDCNMNAPRMNSJ7RD2WA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:38.214886788 (60h34m47.937449888s) [noise=true , address=45.91.116.9:4277 , piece_id=LNGQZ2L7URAJDOGWPQNF554JJ3GAL53UMHKRZQVOW4YQPMCKLYUA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:11:28.856710878 (60h35m38.579273908s) [noise=true , address=104.36.82.52:4115 , piece_id=S6KY6ZLJYXIPQ2YQ4P7EMDGCYJZXNKMDNEGM2YZTWY75HJ2ZB4NQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856757427 (60h35m38.579320457s) [noise=true , address=104.36.82.46:4488 , piece_id=UTRSLBGHKRDZY2LX3OSRCSAV663ZMJ3WZT2V3SIPBHY3D2WG7FKQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: rpc: dial tcp 104.36.82.46:4488: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856831208 (60h35m38.579394238s) [noise=true , address=77.90.31.18:4808 , piece_id=POFPOA5RP3BZPLCEZFQFGCN54DVAVFJUY4BF7HVIJ6LZHI7OTISQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: write tcp 188.166.225.170:50928->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856960185 (60h35m38.579523225s) [noise=true , address=104.36.82.44:4870 , piece_id=IPTBCOOX6CY2LVIBJQAXZGZLQREVQZOEZRJHAQUHHBGNUFVDQ4FQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: rpc: dial tcp 104.36.82.44:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857005111 (60h35m38.579568191s) [noise=true , address=104.36.82.54:4166 , piece_id=ENGGRHWRI3OMM4BOAOK7JGHXFPHAG34DO6FQ75I2BIFMAAJ2HL3Q, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 188.166.225.170:48336->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857048403 (60h35m38.579611443s) [noise=true , address=104.36.82.38:4689 , piece_id=EB6WFS7RTJAYPH2OC4BOBVD3JXLVTBZTTK3YDWB77YJP6PEZJFHA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:104.36.82.38:4689): protocol: write tcp 188.166.225.170:52234->104.36.82.38:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857114791 (60h35m38.579677831s) [noise=true , address=104.36.82.47:4533 , piece_id=PMYE3EDT3DHWUD67JYXTVW4EGQ5KB7NOWQJ2YAKNIBO5X42WTPRQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: write tcp 188.166.225.170:44362->104.36.82.47:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857138366 (60h35m38.579701406s) [seg_pos={0 0}, upload=58101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:11:29.085947751 (60h35m38.808510791s) [upload=58101]: Done uploading segments... Jul 27 00:11:29.085980072 (60h35m38.808543133s) [upload=58101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230977779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XXX.....XXX..X.....X......X...XX..XX.X.XX...X...............] duration: 18.044760268s FAIL (2025-07-27 00:11:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.044788652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.874451351s FAIL (2025-07-27 00:10:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.874501938s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX..] duration: 7.088718198s FAIL (2025-07-27 00:10:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.088741593s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X.] duration: 7.624144686s FAIL (2025-07-27 00:10:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== }, upload=58077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: rpc: dial tcp 104.36.82.3:4765: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.749291092 (60h33m8.471854132s) [seg_pos={0 0}, upload=58077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:08:58.749388238 (60h33m8.471951278s) [upload=58077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:08:58.749395502 (60h33m8.471958542s) [upload=58077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:08:59.071572826 (60h33m8.794135866s) [upload=58077]: Done uploading segments... Jul 27 00:08:59.071590289 (60h33m8.794153319s) [upload=58077]: Upload finished. err: <nil> Jul 27 00:09:23.191431943 (60h33m32.913994983s) [upload=58085]: Starting upload Jul 27 00:09:23.191484463 (60h33m32.914047503s) [upload=58085]: Done starting upload Jul 27 00:09:23.191558114 (60h33m32.914121154s) [upload=58085]: Uploading segments... Jul 27 00:09:23.191660350 (60h33m32.91422345s) [upload=58085]: Got next segment. Inline: false Jul 27 00:09:23.191695326 (60h33m32.914258356s) [upload=58085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:09:23.191699464 (60h33m32.914262504s) [upload=58085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:23.424088299 (60h33m33.146651339s) [seg_pos={0 0}, upload=58085]: Begin upload segment... Jul 27 00:09:23.424132493 (60h33m33.146695533s) [seg_pos={0 0}, upload=58085]: Done begin upload segment. Jul 27 00:09:23.424154705 (60h33m33.146717746s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424193240 (60h33m33.14675628s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424508462 (60h33m33.147071503s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424731339 (60h33m33.147294389s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424777367 (60h33m33.147340397s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425039278 (60h33m33.147602318s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425091207 (60h33m33.147654247s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425286531 (60h33m33.147849571s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425540938 (60h33m33.148103978s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425552049 (60h33m33.148115089s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425763755 (60h33m33.148326785s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425836293 (60h33m33.148399333s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426064771 (60h33m33.148627811s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426081131 (60h33m33.148644171s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426305661 (60h33m33.148868701s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426336901 (60h33m33.148899941s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:25.166110004 (60h33m34.888673044s) [upload=58085]: Next returned nil segment Jul 27 00:09:25.166138388 (60h33m34.888701428s) [upload=58085]: Waiting for error group managing segments... Jul 27 00:09:25.453532572 (60h33m35.176095613s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.457229505 (60h33m35.179792545s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.526191879 (60h33m35.248754919s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.527573101 (60h33m35.250136142s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.531417748 (60h33m35.253980788s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.542134953 (60h33m35.264697993s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.550081446 (60h33m35.272644486s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.608111033 (60h33m35.330674073s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.614659251 (60h33m35.337222291s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.731328030 (60h33m35.453891081s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.884837420 (60h33m36.60740046s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.902326182 (60h33m36.624889222s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:52.543670387 (60h34m2.266233467s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41546->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543696337 (60h34m2.266259427s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543714811 (60h34m2.266277881s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41130->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543776821 (60h34m2.266339921s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:49080->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543813270 (60h34m2.26637632s) [seg_pos={0 0}, upload=58085]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:52.770059330 (60h34m2.49262237s) [upload=58085]: Done uploading segments... Jul 27 00:09:52.770089268 (60h34m2.492652308s) [upload=58085]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.624172981s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXX] duration: 30.000458881s FAIL (2025-07-27 00:10:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000469311s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.25568657s FAIL (2025-07-27 00:10:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 15762s) [upload=58086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:24.686101576 (60h33m34.408664616s) [seg_pos={0 0}, upload=58086]: Begin upload segment... Jul 27 00:09:24.686160258 (60h33m34.408723288s) [seg_pos={0 0}, upload=58086]: Done begin upload segment. Jul 27 00:09:24.686189354 (60h33m34.408752394s) [noise=true , address=79.127.205.246:4260 , piece_id=QEI2FBYJJGMY6FHYU25QOFTXIOQRWTVXNVKCNPVIQHVMON4UKFFQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686197971 (60h33m34.408761001s) [noise=true , address=156.146.43.225:4448 , piece_id=YLGP74SY5PSZETOZDSDR6KNKU2XQFQ5OQ6NUT465E2PPRTL6VLFA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686587656 (60h33m34.409150697s) [noise=true , address=77.90.31.15:4717 , piece_id=YUL6OKG4DOZ5LL5SAB3XLZT3IWBAJDBTWDBVYLOC2ALMZPL55HGQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686610470 (60h33m34.40917351s) [noise=true , address=104.36.82.14:4249 , piece_id=BSJHZEKSG33WSWXWVM5AHWJDEIVI5D7GMF4M2FQBIJ4IE3JLTO5Q, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686910794 (60h33m34.409473824s) [noise=true , address=79.127.201.215:4357 , piece_id=ZKIJASB57WEYXHB46PT2UHHBGTSFFWJZHAXZFUTLDRFHZUW4YVFQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686971751 (60h33m34.409534791s) [noise=true , address=77.90.31.19:4847 , piece_id=C4X7BU57535CR7YSGN73C3YF4SX6A6RUEI2H2YVXHDIXRUZMQNCA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687216329 (60h33m34.40977936s) [noise=true , address=104.36.82.55:4217 , piece_id=5DY4Q3OU7BT7LEQIPIA22SPCIKLMH2CAJSBU366DXOGGANCV5GZQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687286513 (60h33m34.409849553s) [noise=true , address=79.127.219.47:4048 , piece_id=ZMCODSORUV6Y37UWYC7PUPVTURVVCP4MZ74HSAFALKQWOLJPPN3A, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687496445 (60h33m34.410059485s) [noise=true , address=77.90.31.10:4528 , piece_id=XMNFWOI55V7CIKQFBIBSTTSVWW4UAN5GMOQRDTUMZIAHRYEVDBDA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687753146 (60h33m34.410316186s) [noise=true , address=79.127.219.38:4004 , piece_id=TG7AXUQ7XTLFENB4RFNFAMYZUY2DCR7MSGSMOZK5VEJA5ZUCXDGA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687991753 (60h33m34.410554783s) [noise=true , address=79.127.205.234:4597 , piece_id=KMQAEF663QK2V5LR2ZML6EE7AAUNH72PW2HQXLPWKYG5C4WFPVAA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.688301886 (60h33m34.410864926s) [noise=true , address=79.127.219.48:4074 , piece_id=5GU4SGJYKYULA53USXR2CJFLBGPMVAZ3AD3ROEKQDXOMMINN7GQA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.688518040 (60h33m34.41108108s) [noise=true , address=104.36.82.46:4488 , piece_id=6MQUXWLTRNTKO6XWRF2YC5G4X4BUIBZ75FMK72XJ34KHV3EQLCAA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.688777977 (60h33m34.411341007s) [noise=true , address=64.49.13.138:4488 , piece_id=TV2WT4RXACQVFVAIS3EIR5V3KNBA5ZWQQ7M2WWOKYC5WYQ2DCZFA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.688995473 (60h33m34.411558513s) [noise=true , address=79.127.205.245:4242 , piece_id=VPKPS44T6WPAS6ITZKVHWSLGNE6VO5SUL6XDMQY4H4SHTM7JAEKA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.689256663 (60h33m34.411819703s) [noise=true , address=79.127.205.235:4689 , piece_id=X747T7GUJFOCFAF2RAIDQVE6XEZXU753JF6YM5474RJKBS5TITVA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.689466354 (60h33m34.412029394s) [noise=true , address=104.36.82.56:4259 , piece_id=GUP64TUWZ4DCOZAA6FN2ASOH7QF2AZVFNVN2TJMVO7KI7XJ4CKIQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.689707966 (60h33m34.412270996s) [noise=true , address=104.36.82.19:4427 , piece_id=VKTF65IPBPUCSNCL5HLEFSLBWC3OSSAOEVTJU2RI7T26DX6UAYQA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:25.792629708 (60h33m35.515192748s) [upload=58086]: Next returned nil segment Jul 27 00:09:25.793502126 (60h33m35.516065156s) [upload=58086]: Waiting for error group managing segments... Jul 27 00:09:25.805481586 (60h33m35.528044626s) [noise=true , address=79.127.219.48:4074 , piece_id=5GU4SGJYKYULA53USXR2CJFLBGPMVAZ3AD3ROEKQDXOMMINN7GQA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:127YpFQAmSFbgY2cmAEwcCPDUhg9AnRHS3SkcafEaQeSGVUwVdj, address:79.127.219.48:4074): protocol: storage node overloaded, request limit: 2000 Jul 27 00:09:26.102507057 (60h33m35.825070097s) [noise=true , address=79.127.205.245:4242 , piece_id=VPKPS44T6WPAS6ITZKVHWSLGNE6VO5SUL6XDMQY4H4SHTM7JAEKA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.108799727 (60h33m35.831362777s) [noise=true , address=79.127.205.235:4689 , piece_id=X747T7GUJFOCFAF2RAIDQVE6XEZXU753JF6YM5474RJKBS5TITVA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.123596598 (60h33m35.846159638s) [noise=true , address=79.127.205.246:4260 , piece_id=QEI2FBYJJGMY6FHYU25QOFTXIOQRWTVXNVKCNPVIQHVMON4UKFFQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.160404912 (60h33m35.882967952s) [noise=true , address=79.127.205.234:4597 , piece_id=KMQAEF663QK2V5LR2ZML6EE7AAUNH72PW2HQXLPWKYG5C4WFPVAA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.692108297 (60h33m36.414671337s) [noise=true , address=79.127.219.38:4004 , piece_id=TG7AXUQ7XTLFENB4RFNFAMYZUY2DCR7MSGSMOZK5VEJA5ZUCXDGA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.874541202 (60h33m36.597104232s) [noise=true , address=77.90.31.10:4528 , piece_id=XMNFWOI55V7CIKQFBIBSTTSVWW4UAN5GMOQRDTUMZIAHRYEVDBDA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:27.819559289 (60h33m37.542122329s) [noise=true , address=79.127.219.47:4048 , piece_id=ZMCODSORUV6Y37UWYC7PUPVTURVVCP4MZ74HSAFALKQWOLJPPN3A, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:28.856775588 (60h33m38.579338628s) [noise=true , address=64.49.13.138:4488 , piece_id=TV2WT4RXACQVFVAIS3EIR5V3KNBA5ZWQQ7M2WWOKYC5WYQ2DCZFA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:29.980372795 (60h33m39.702935835s) [noise=true , address=156.146.43.225:4448 , piece_id=YLGP74SY5PSZETOZDSDR6KNKU2XQFQ5OQ6NUT465E2PPRTL6VLFA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:43.674892227 (60h33m53.397455267s) [noise=true , address=77.90.31.19:4847 , piece_id=C4X7BU57535CR7YSGN73C3YF4SX6A6RUEI2H2YVXHDIXRUZMQNCA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:10:01.216970223 (60h34m10.939533264s) [noise=true , address=79.127.201.215:4357 , piece_id=ZKIJASB57WEYXHB46PT2UHHBGTSFFWJZHAXZFUTLDRFHZUW4YVFQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:10:23.570404312 (60h34m33.292967352s) [noise=true , address=104.36.82.56:4259 , piece_id=GUP64TUWZ4DCOZAA6FN2ASOH7QF2AZVFNVN2TJMVO7KI7XJ4CKIQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: rpc: dial tcp 104.36.82.56:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570447816 (60h34m33.293010856s) [noise=true , address=104.36.82.55:4217 , piece_id=5DY4Q3OU7BT7LEQIPIA22SPCIKLMH2CAJSBU366DXOGGANCV5GZQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: rpc: dial tcp 104.36.82.55:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570480628 (60h34m33.293043668s) [noise=true , address=104.36.82.46:4488 , piece_id=6MQUXWLTRNTKO6XWRF2YC5G4X4BUIBZ75FMK72XJ34KHV3EQLCAA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: rpc: dial tcp 104.36.82.46:4488: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570492442 (60h34m33.293055482s) [noise=true , address=104.36.82.14:4249 , piece_id=BSJHZEKSG33WSWXWVM5AHWJDEIVI5D7GMF4M2FQBIJ4IE3JLTO5Q, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:1GnfZTc54L9eRyjCUwQuaFcG6FjsUDjDwT3Gx4zoNTQrvP158L, address:104.36.82.14:4249): protocol: rpc: dial tcp 104.36.82.14:4249: i/o timeout; context deadline exceeded Jul 27 00:10:23.570511227 (60h34m33.293074267s) [noise=true , address=104.36.82.19:4427 , piece_id=VKTF65IPBPUCSNCL5HLEFSLBWC3OSSAOEVTJU2RI7T26DX6UAYQA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570546214 (60h34m33.293109255s) [noise=true , address=77.90.31.15:4717 , piece_id=YUL6OKG4DOZ5LL5SAB3XLZT3IWBAJDBTWDBVYLOC2ALMZPL55HGQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 188.166.225.170:53434->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570566783 (60h34m33.293129823s) [seg_pos={0 0}, upload=58086]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:10:23.823831128 (60h34m33.546394168s) [upload=58086]: Done uploading segments... Jul 27 00:10:23.823856045 (60h34m33.546419085s) [upload=58086]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.255703863s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..............] duration: 30.211471675s FAIL (2025-07-27 00:10:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.211485181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX.] duration: 6.993072152s FAIL (2025-07-27 00:10:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.993097601s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XX] duration: 30.000893536s FAIL (2025-07-27 00:09:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000932181s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X] duration: 30.227924289s FAIL (2025-07-27 00:09:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== }, upload=58077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: rpc: dial tcp 104.36.82.3:4765: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.749291092 (60h33m8.471854132s) [seg_pos={0 0}, upload=58077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:08:58.749388238 (60h33m8.471951278s) [upload=58077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:08:58.749395502 (60h33m8.471958542s) [upload=58077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:08:59.071572826 (60h33m8.794135866s) [upload=58077]: Done uploading segments... Jul 27 00:08:59.071590289 (60h33m8.794153319s) [upload=58077]: Upload finished. err: <nil> Jul 27 00:09:23.191431943 (60h33m32.913994983s) [upload=58085]: Starting upload Jul 27 00:09:23.191484463 (60h33m32.914047503s) [upload=58085]: Done starting upload Jul 27 00:09:23.191558114 (60h33m32.914121154s) [upload=58085]: Uploading segments... Jul 27 00:09:23.191660350 (60h33m32.91422345s) [upload=58085]: Got next segment. Inline: false Jul 27 00:09:23.191695326 (60h33m32.914258356s) [upload=58085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:09:23.191699464 (60h33m32.914262504s) [upload=58085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:23.424088299 (60h33m33.146651339s) [seg_pos={0 0}, upload=58085]: Begin upload segment... Jul 27 00:09:23.424132493 (60h33m33.146695533s) [seg_pos={0 0}, upload=58085]: Done begin upload segment. Jul 27 00:09:23.424154705 (60h33m33.146717746s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424193240 (60h33m33.14675628s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424508462 (60h33m33.147071503s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424731339 (60h33m33.147294389s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424777367 (60h33m33.147340397s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425039278 (60h33m33.147602318s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425091207 (60h33m33.147654247s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425286531 (60h33m33.147849571s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425540938 (60h33m33.148103978s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425552049 (60h33m33.148115089s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425763755 (60h33m33.148326785s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425836293 (60h33m33.148399333s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426064771 (60h33m33.148627811s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426081131 (60h33m33.148644171s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426305661 (60h33m33.148868701s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426336901 (60h33m33.148899941s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:25.166110004 (60h33m34.888673044s) [upload=58085]: Next returned nil segment Jul 27 00:09:25.166138388 (60h33m34.888701428s) [upload=58085]: Waiting for error group managing segments... Jul 27 00:09:25.453532572 (60h33m35.176095613s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.457229505 (60h33m35.179792545s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.526191879 (60h33m35.248754919s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.527573101 (60h33m35.250136142s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.531417748 (60h33m35.253980788s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.542134953 (60h33m35.264697993s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.550081446 (60h33m35.272644486s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.608111033 (60h33m35.330674073s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.614659251 (60h33m35.337222291s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.731328030 (60h33m35.453891081s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.884837420 (60h33m36.60740046s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.902326182 (60h33m36.624889222s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:52.543670387 (60h34m2.266233467s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41546->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543696337 (60h34m2.266259427s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543714811 (60h34m2.266277881s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41130->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543776821 (60h34m2.266339921s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:49080->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543813270 (60h34m2.26637632s) [seg_pos={0 0}, upload=58085]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:52.770059330 (60h34m2.49262237s) [upload=58085]: Done uploading segments... Jul 27 00:09:52.770089268 (60h34m2.492652308s) [upload=58085]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.227946281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.847118933s FAIL (2025-07-27 00:09:47 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.847148731s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.............] duration: 52.860733119s FAIL (2025-07-27 00:09:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.860746493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX] duration: 30.264421826s FAIL (2025-07-27 00:09:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.264443898s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234980967s FAIL (2025-07-27 00:09:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== , piece_id=HUUZUMK2MXON23DYDTI5SAVL2PDGQCZJBFOSBBSVY3I5IXCRBXAA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191468653 (60h32m33.914031693s) [noise=true , address=79.127.219.38:4003 , piece_id=HGV4JIAWMIF52L6BMBE2JTBDSZXRKDPBRCIBFS72SDIKUL5X4QCQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191592921 (60h32m33.914155961s) [noise=true , address=45.91.116.9:4277 , piece_id=VI4FPX34RBSQBZVYJG6JEHWH3VDIZFD36TSEYKUWBC5MZTSCFD4A, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191758949 (60h32m33.914322019s) [noise=true , address=77.90.31.12:4608 , piece_id=PNSNCD462QGF44F57B5ABKWNAFXNXRRCIJOUPLQWWYR5UIYC3LQA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191896712 (60h32m33.914459752s) [noise=true , address=104.36.82.50:4016 , piece_id=WDZYETH3W6LW2T6UWYTQMN6SFLGHIAJWEDRAD2P3AE4A2UMNCPOA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192077898 (60h32m33.914640939s) [noise=true , address=79.127.220.101:4041 , piece_id=EMIQWCARLYTXACBD6O6DZH77RLYWX7EKA5PMFVNCJVQLCFW7WLFA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192227135 (60h32m33.914790175s) [noise=true , address=104.36.82.47:4532 , piece_id=BV2SJ3WCFJVTTO3A5QMITXSZVAMCQTKHJZOQQEQWVE5B4FUCHCJQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192366661 (60h32m33.914929702s) [noise=true , address=79.127.205.245:4250 , piece_id=KVMWKUPRKG2AD5YD2DLDBASPNKQWX3DVK4HPGJRTQE4HZEF36SZA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192522870 (60h32m33.91508591s) [noise=true , address=79.127.219.47:4048 , piece_id=LWIVS27EL4PHHBJ7W44KW3LNXSLK3VLXLGACZPIQZMIQVRLZ5HCA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192645184 (60h32m33.915208244s) [noise=true , address=79.127.205.235:4680 , piece_id=5SEJDTYH3PQAZSRZ3BNCILT4SPTDBN6F2I5CBM7XCF5HDFPYUU2A, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192798868 (60h32m33.915361908s) [noise=true , address=104.36.82.54:4192 , piece_id=ERPUZFYA2ND4TU5NBQPX22UW426C4VSOW2Z7TJBNYIMTAJUVVP4Q, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192909720 (60h32m33.91547276s) [noise=true , address=104.36.82.41:4783 , piece_id=FSSYD2VORTCAASQAK62RV4RFU5XFRJXNT4LTSRFQDCC5HJCNOUKQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193028888 (60h32m33.915591928s) [noise=true , address=104.36.82.46:4491 , piece_id=6ACSWEXX2EOI6Y2JHZIDTJ22AKUTUSZZUR7MCY4WE5XFEOWKHMQA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193189866 (60h32m33.915752906s) [noise=true , address=79.127.201.215:4356 , piece_id=DQHKW2AN3YEYLHOGW6JPUVLP4JWLCK5NFZ5AKIQRDQ3ZQZFOZYFQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193269258 (60h32m33.915832298s) [noise=true , address=45.91.116.10:4093 , piece_id=3AHXMLVONPDEVJLF7M7FKS4TWOU5HD3GJJIBHB5K5YQKC7O4A6HQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193446668 (60h32m33.916009708s) [noise=true , address=79.127.219.48:4087 , piece_id=RDDEOGNCM5DDUBZS7HPVJPKBK6GWKOHX7NCCS5GXROE3MXDUY2SA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193675024 (60h32m33.916238064s) [noise=true , address=156.146.43.225:4434 , piece_id=M4DFDGN23AS7FY326EBM57YNL3Y552CWGHOEDVUYLB5L25IECD7Q, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:25.135671862 (60h32m34.858234902s) [noise=true , address=79.127.220.101:4041 , piece_id=EMIQWCARLYTXACBD6O6DZH77RLYWX7EKA5PMFVNCJVQLCFW7WLFA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:08:25.229683572 (60h32m34.952246612s) [noise=true , address=79.127.219.47:4048 , piece_id=LWIVS27EL4PHHBJ7W44KW3LNXSLK3VLXLGACZPIQZMIQVRLZ5HCA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:08:25.308928970 (60h32m35.03149201s) [upload=58070]: Next returned nil segment Jul 27 00:08:25.308950642 (60h32m35.031513682s) [upload=58070]: Waiting for error group managing segments... Jul 27 00:08:25.596587471 (60h32m35.319150501s) [noise=true , address=79.127.205.235:4680 , piece_id=5SEJDTYH3PQAZSRZ3BNCILT4SPTDBN6F2I5CBM7XCF5HDFPYUU2A, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:25.596694865 (60h32m35.319257905s) [noise=true , address=79.127.205.245:4250 , piece_id=KVMWKUPRKG2AD5YD2DLDBASPNKQWX3DVK4HPGJRTQE4HZEF36SZA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:26.179102912 (60h32m35.901665952s) [noise=true , address=79.127.219.38:4003 , piece_id=HGV4JIAWMIF52L6BMBE2JTBDSZXRKDPBRCIBFS72SDIKUL5X4QCQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:26.303026391 (60h32m36.025589431s) [noise=true , address=79.127.219.48:4087 , piece_id=RDDEOGNCM5DDUBZS7HPVJPKBK6GWKOHX7NCCS5GXROE3MXDUY2SA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:29.930039220 (60h32m39.65260226s) [noise=true , address=45.91.116.10:4093 , piece_id=3AHXMLVONPDEVJLF7M7FKS4TWOU5HD3GJJIBHB5K5YQKC7O4A6HQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:29.939541440 (60h32m39.66210449s) [noise=true , address=77.90.31.12:4608 , piece_id=PNSNCD462QGF44F57B5ABKWNAFXNXRRCIJOUPLQWWYR5UIYC3LQA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:30.410036577 (60h32m40.132599657s) [noise=true , address=156.146.43.225:4434 , piece_id=M4DFDGN23AS7FY326EBM57YNL3Y552CWGHOEDVUYLB5L25IECD7Q, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:35.944972414 (60h32m45.667535484s) [noise=true , address=45.91.116.9:4277 , piece_id=VI4FPX34RBSQBZVYJG6JEHWH3VDIZFD36TSEYKUWBC5MZTSCFD4A, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:59.456991076 (60h33m9.179554116s) [noise=true , address=79.127.201.215:4356 , piece_id=DQHKW2AN3YEYLHOGW6JPUVLP4JWLCK5NFZ5AKIQRDQ3ZQZFOZYFQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:09:23.094493363 (60h33m32.817056393s) [noise=true , address=104.36.82.54:4192 , piece_id=ERPUZFYA2ND4TU5NBQPX22UW426C4VSOW2Z7TJBNYIMTAJUVVP4Q, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: rpc: dial tcp 104.36.82.54:4192: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094541304 (60h33m32.817104344s) [noise=true , address=104.36.82.50:4016 , piece_id=WDZYETH3W6LW2T6UWYTQMN6SFLGHIAJWEDRAD2P3AE4A2UMNCPOA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR, address:104.36.82.50:4016): protocol: write tcp 188.166.225.170:59734->104.36.82.50:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094859924 (60h33m32.817422994s) [noise=true , address=104.36.82.47:4532 , piece_id=BV2SJ3WCFJVTTO3A5QMITXSZVAMCQTKHJZOQQEQWVE5B4FUCHCJQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: write tcp 188.166.225.170:41088->104.36.82.47:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094920129 (60h33m32.817483219s) [noise=true , address=104.36.82.46:4491 , piece_id=6ACSWEXX2EOI6Y2JHZIDTJ22AKUTUSZZUR7MCY4WE5XFEOWKHMQA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX, address:104.36.82.46:4491): protocol: write tcp 188.166.225.170:51990->104.36.82.46:4491: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095027074 (60h33m32.817590154s) [noise=true , address=104.36.82.55:4213 , piece_id=XAY5SRUIMYDFPKPWZTBSVHCNFSAVMOPKOIB7ZZ6AJEPN7WNB4AAA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: rpc: dial tcp 104.36.82.55:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095099132 (60h33m32.817662172s) [noise=true , address=104.36.82.41:4783 , piece_id=FSSYD2VORTCAASQAK62RV4RFU5XFRJXNT4LTSRFQDCC5HJCNOUKQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: write tcp 188.166.225.170:54126->104.36.82.41:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095202749 (60h33m32.817765819s) [noise=true , address=104.36.82.10:4094 , piece_id=HUUZUMK2MXON23DYDTI5SAVL2PDGQCZJBFOSBBSVY3I5IXCRBXAA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:124zBCfZBTW69BSaGRzUyjWUBjSJhaLMZXQYnqMPGSKL4EX6HVV, address:104.36.82.10:4094): protocol: write tcp 188.166.225.170:48722->104.36.82.10:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095250731 (60h33m32.817813791s) [seg_pos={0 0}, upload=58070]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:09:23.327131253 (60h33m33.049694293s) [upload=58070]: Done uploading segments... Jul 27 00:09:23.327157522 (60h33m33.049720562s) [upload=58070]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.234997437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...X] duration: 30.000883423s FAIL (2025-07-27 00:09:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000905465s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..] duration: 7.289167177s FAIL (2025-07-27 00:09:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:07:11.755040247 (60h31m21.477603287s) [upload=58048]: Starting upload Jul 27 00:07:11.755105291 (60h31m21.477668331s) [upload=58048]: Done starting upload Jul 27 00:07:11.755198160 (60h31m21.47776122s) [upload=58048]: Uploading segments... Jul 27 00:07:11.755291858 (60h31m21.477854969s) [upload=58048]: Got next segment. Inline: false Jul 27 00:07:11.755341433 (60h31m21.477904474s) [upload=58048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:11.755345661 (60h31m21.477908701s) [upload=58048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:11.990165825 (60h31m21.712728865s) [seg_pos={0 0}, upload=58048]: Begin upload segment... Jul 27 00:07:11.990211012 (60h31m21.712774052s) [seg_pos={0 0}, upload=58048]: Done begin upload segment. Jul 27 00:07:11.990221621 (60h31m21.712784652s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990230568 (60h31m21.712793608s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990633510 (60h31m21.71319655s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990967869 (60h31m21.713530899s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.991721360 (60h31m21.7142844s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992179327 (60h31m21.714742367s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992587718 (60h31m21.715150759s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992994487 (60h31m21.715557527s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993531444 (60h31m21.716094484s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993948862 (60h31m21.716511902s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.994373135 (60h31m21.716936175s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995537432 (60h31m21.718100472s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995812709 (60h31m21.718375749s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996213716 (60h31m21.718776756s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996482962 (60h31m21.719046002s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996721988 (60h31m21.719285028s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:13.734955056 (60h31m23.457518096s) [upload=58048]: Next returned nil segment Jul 27 00:07:13.734972579 (60h31m23.457535619s) [upload=58048]: Waiting for error group managing segments... Jul 27 00:07:14.013610340 (60h31m23.73617338s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018322977 (60h31m23.740886027s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018799488 (60h31m23.741362528s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.089681915 (60h31m23.812244955s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.098404755 (60h31m23.820967795s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.106109907 (60h31m23.828672947s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.112275553 (60h31m23.834838583s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.113721300 (60h31m23.83628433s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.117463299 (60h31m23.840026339s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.178263677 (60h31m23.900826717s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.464873981 (60h31m24.187437021s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:15.435585766 (60h31m25.158148796s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:41.107251487 (60h31m50.829814527s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107464594 (60h31m50.830027664s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:57142->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107538756 (60h31m50.830101827s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38974->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107575747 (60h31m50.830138787s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:07:41.107609272 (60h31m50.830172302s) [seg_pos={0 0}, upload=58048]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:07:41.340882517 (60h31m51.063445547s) [upload=58048]: Done uploading segments... Jul 27 00:07:41.340925319 (60h31m51.063488369s) [upload=58048]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.289196683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..XXXX.....XXX..X.....X......X...XX..XX.X.XX...X............] duration: 47.107254625s FAIL (2025-07-27 00:08:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.107266408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.872802594s FAIL (2025-07-27 00:08:49 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.872831339s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXX] duration: 30.239811509s FAIL (2025-07-27 00:08:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.239842867s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.817189051s FAIL (2025-07-27 00:08:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.81722028s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.244427209s FAIL (2025-07-27 00:08:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ddress=64.49.13.139:4478 , piece_id=6UJX4IIE6CYNLDMRUJ6BHLD65WT3UXZFVNRPD6MEVIUHPR6GXSAA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727061052 (60h31m33.449624122s) [noise=true , address=79.127.220.97:4361 , piece_id=TYJP6IPDMMHMERFCFJNL4NXJZZCG2UYNS2KB3JPGXWXEPINXJ2PA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727196460 (60h31m33.44975955s) [noise=true , address=77.90.31.15:4726 , piece_id=7KKKQMUN7BQHWDMCJTPGNKDKPFGUR33HY4CBCHH343ICI467ROXA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727682550 (60h31m33.45024561s) [noise=true , address=104.36.82.54:4191 , piece_id=A7MNSY6RGTLUT6V46SLEEAQOCPXDRY5QFXBCQOAZ2KHUBQY73SWA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728099728 (60h31m33.450662758s) [noise=true , address=104.36.82.46:4493 , piece_id=KTQPHH3XGIMXINOQRHSVA7ZY2VBOHTFI6LLHGUDBO5X5JBKVJE6A, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728106522 (60h31m33.450669562s) [noise=true , address=79.127.205.235:4687 , piece_id=3J6XO5R7RAKTZZVUSTCQ5S545NDCFCLXO3PBFSHXFWSZANLNDLAA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728548447 (60h31m33.451111487s) [noise=true , address=104.36.82.55:4204 , piece_id=VYOPSSZNCYIO2NE5GOSKVWOEDXSNCMXDLIEHMHSJYXF6KQHAE6QA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.729766277 (60h31m33.452329317s) [noise=true , address=79.127.219.38:4003 , piece_id=N5R7ILGBGWODMYVA72UKNSBLHO2ZCZOHPVGIKSSRILX2TDNAK6QA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.730872413 (60h31m33.453435453s) [noise=true , address=104.36.82.16:4300 , piece_id=SW3L2SBYEEUMIEPT6BNUUBGYLBGZW4CB3KZUWVER2MNHZ3GH6HEA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.731313136 (60h31m33.453876177s) [noise=true , address=77.90.31.16:4737 , piece_id=H3GQJUAD3HGZLR4UFSVWNQDWWYFQGABYWJYWQPZULNFETEN2V6MA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.731703463 (60h31m33.454266503s) [noise=true , address=104.36.82.53:4142 , piece_id=WMZCU4FYC4KXVD5TO7PQ5S45NSSJL4IN45UNRBZTUZ5N6MBA35WA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732105703 (60h31m33.454668743s) [noise=true , address=79.127.205.234:4610 , piece_id=GBFRWZZDAX2SUH55K7SHR4MBTV5ADHFH26JOR2F7D5R5VWVFKVTA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732370520 (60h31m33.45493357s) [noise=true , address=79.127.219.47:4048 , piece_id=DS26KSX25RBT3DEEKXDFNXCROHU2AINQKMMS5TN6DJMTNVVUA56Q, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732647630 (60h31m33.45521067s) [noise=true , address=79.127.205.246:4270 , piece_id=AY4DVMJGAVSKY5C2ZP4WAAQZPRA2KNHP6HBSTJHY6OFL3MT5NCNA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732921374 (60h31m33.455484414s) [noise=true , address=45.91.116.10:4093 , piece_id=USC6TAFNA35EBKBQLDEJRSIYUESRZWSEI6SMICMWISZY7U7R7RHA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.733277195 (60h31m33.455840225s) [noise=true , address=79.127.201.215:4355 , piece_id=RV2YPT7IJ5TRCXJMXCWBZTHS43ME42WUFRYUF3LLD6FNKI2BEYIA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:24.820008849 (60h31m34.542571889s) [upload=58052]: Next returned nil segment Jul 27 00:07:24.820025310 (60h31m34.542588351s) [upload=58052]: Waiting for error group managing segments... Jul 27 00:07:24.887653151 (60h31m34.610216191s) [noise=true , address=79.127.219.47:4048 , piece_id=DS26KSX25RBT3DEEKXDFNXCROHU2AINQKMMS5TN6DJMTNVVUA56Q, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:07:25.109410853 (60h31m34.831973893s) [noise=true , address=79.127.205.234:4610 , piece_id=GBFRWZZDAX2SUH55K7SHR4MBTV5ADHFH26JOR2F7D5R5VWVFKVTA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.138673642 (60h31m34.861236682s) [noise=true , address=79.127.205.235:4687 , piece_id=3J6XO5R7RAKTZZVUSTCQ5S545NDCFCLXO3PBFSHXFWSZANLNDLAA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.153927788 (60h31m34.876490828s) [noise=true , address=79.127.205.246:4270 , piece_id=AY4DVMJGAVSKY5C2ZP4WAAQZPRA2KNHP6HBSTJHY6OFL3MT5NCNA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.642669774 (60h31m35.365232814s) [noise=true , address=79.127.220.97:4361 , piece_id=TYJP6IPDMMHMERFCFJNL4NXJZZCG2UYNS2KB3JPGXWXEPINXJ2PA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.719944903 (60h31m35.442507943s) [noise=true , address=79.127.219.38:4003 , piece_id=N5R7ILGBGWODMYVA72UKNSBLHO2ZCZOHPVGIKSSRILX2TDNAK6QA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.757257171 (60h31m35.479820201s) [noise=true , address=77.90.31.16:4737 , piece_id=H3GQJUAD3HGZLR4UFSVWNQDWWYFQGABYWJYWQPZULNFETEN2V6MA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:35.745696491 (60h31m45.468259532s) [noise=true , address=64.49.13.139:4478 , piece_id=6UJX4IIE6CYNLDMRUJ6BHLD65WT3UXZFVNRPD6MEVIUHPR6GXSAA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:40.589949786 (60h31m50.312512837s) [noise=true , address=45.91.116.10:4093 , piece_id=USC6TAFNA35EBKBQLDEJRSIYUESRZWSEI6SMICMWISZY7U7R7RHA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:47.462943792 (60h31m57.185506833s) [noise=true , address=79.127.201.215:4355 , piece_id=RV2YPT7IJ5TRCXJMXCWBZTHS43ME42WUFRYUF3LLD6FNKI2BEYIA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:08:22.624850013 (60h32m32.347413073s) [noise=true , address=77.90.31.15:4726 , piece_id=7KKKQMUN7BQHWDMCJTPGNKDKPFGUR33HY4CBCHH343ICI467ROXA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: write tcp 188.166.225.170:49330->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624893377 (60h32m32.347456417s) [noise=true , address=104.36.82.55:4204 , piece_id=VYOPSSZNCYIO2NE5GOSKVWOEDXSNCMXDLIEHMHSJYXF6KQHAE6QA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:44342->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624933333 (60h32m32.347496373s) [noise=true , address=104.36.82.16:4300 , piece_id=SW3L2SBYEEUMIEPT6BNUUBGYLBGZW4CB3KZUWVER2MNHZ3GH6HEA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg, address:104.36.82.16:4300): protocol: write tcp 188.166.225.170:38544->104.36.82.16:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624963520 (60h32m32.34752656s) [noise=true , address=104.36.82.54:4191 , piece_id=A7MNSY6RGTLUT6V46SLEEAQOCPXDRY5QFXBCQOAZ2KHUBQY73SWA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: write tcp 188.166.225.170:53522->104.36.82.54:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624996474 (60h32m32.347559554s) [noise=true , address=104.36.82.46:4493 , piece_id=KTQPHH3XGIMXINOQRHSVA7ZY2VBOHTFI6LLHGUDBO5X5JBKVJE6A, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 188.166.225.170:59700->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625029076 (60h32m32.347592277s) [noise=true , address=104.36.82.47:4526 , piece_id=6DBPGXO5PJCFQ7PKXOZK2IM6AELQQ7IC2MKVDWPPASOYSKX7PMPQ, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:48354->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625062891 (60h32m32.347626101s) [noise=true , address=104.36.82.53:4142 , piece_id=WMZCU4FYC4KXVD5TO7PQ5S45NSSJL4IN45UNRBZTUZ5N6MBA35WA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: write tcp 188.166.225.170:54566->104.36.82.53:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625098859 (60h32m32.347661899s) [noise=true , address=104.36.82.2:4736 , piece_id=ICMVAPJZ3HNI6AGWXV6NYKA5Y7W7IFOT2IFVGVWSNEQTNGWTDIGQ, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:104.36.82.2:4736): protocol: write tcp 188.166.225.170:57946->104.36.82.2:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625117215 (60h32m32.347680255s) [seg_pos={0 0}, upload=58052]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:08:22.867247115 (60h32m32.589810155s) [upload=58052]: Done uploading segments... Jul 27 00:08:22.867270961 (60h32m32.589834001s) [upload=58052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.24443828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...XXXX.....XXX..X.....X......X...XX..XX.X.XX...X...........] duration: 50.659346226s FAIL (2025-07-27 00:08:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.659380051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...] duration: 7.359180024s FAIL (2025-07-27 00:08:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.359199812s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.631167904s FAIL (2025-07-27 00:07:51 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.631192571s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX.] duration: 7.190647044s FAIL (2025-07-27 00:07:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:07:11.755040247 (60h31m21.477603287s) [upload=58048]: Starting upload Jul 27 00:07:11.755105291 (60h31m21.477668331s) [upload=58048]: Done starting upload Jul 27 00:07:11.755198160 (60h31m21.47776122s) [upload=58048]: Uploading segments... Jul 27 00:07:11.755291858 (60h31m21.477854969s) [upload=58048]: Got next segment. Inline: false Jul 27 00:07:11.755341433 (60h31m21.477904474s) [upload=58048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:11.755345661 (60h31m21.477908701s) [upload=58048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:11.990165825 (60h31m21.712728865s) [seg_pos={0 0}, upload=58048]: Begin upload segment... Jul 27 00:07:11.990211012 (60h31m21.712774052s) [seg_pos={0 0}, upload=58048]: Done begin upload segment. Jul 27 00:07:11.990221621 (60h31m21.712784652s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990230568 (60h31m21.712793608s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990633510 (60h31m21.71319655s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990967869 (60h31m21.713530899s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.991721360 (60h31m21.7142844s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992179327 (60h31m21.714742367s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992587718 (60h31m21.715150759s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992994487 (60h31m21.715557527s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993531444 (60h31m21.716094484s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993948862 (60h31m21.716511902s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.994373135 (60h31m21.716936175s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995537432 (60h31m21.718100472s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995812709 (60h31m21.718375749s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996213716 (60h31m21.718776756s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996482962 (60h31m21.719046002s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996721988 (60h31m21.719285028s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:13.734955056 (60h31m23.457518096s) [upload=58048]: Next returned nil segment Jul 27 00:07:13.734972579 (60h31m23.457535619s) [upload=58048]: Waiting for error group managing segments... Jul 27 00:07:14.013610340 (60h31m23.73617338s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018322977 (60h31m23.740886027s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018799488 (60h31m23.741362528s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.089681915 (60h31m23.812244955s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.098404755 (60h31m23.820967795s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.106109907 (60h31m23.828672947s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.112275553 (60h31m23.834838583s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.113721300 (60h31m23.83628433s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.117463299 (60h31m23.840026339s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.178263677 (60h31m23.900826717s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.464873981 (60h31m24.187437021s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:15.435585766 (60h31m25.158148796s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:41.107251487 (60h31m50.829814527s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107464594 (60h31m50.830027664s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:57142->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107538756 (60h31m50.830101827s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38974->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107575747 (60h31m50.830138787s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:07:41.107609272 (60h31m50.830172302s) [seg_pos={0 0}, upload=58048]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:07:41.340882517 (60h31m51.063445547s) [upload=58048]: Done uploading segments... Jul 27 00:07:41.340925319 (60h31m51.063488369s) [upload=58048]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.190664127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX] duration: 30.235801722s FAIL (2025-07-27 00:07:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:07:11.755040247 (60h31m21.477603287s) [upload=58048]: Starting upload Jul 27 00:07:11.755105291 (60h31m21.477668331s) [upload=58048]: Done starting upload Jul 27 00:07:11.755198160 (60h31m21.47776122s) [upload=58048]: Uploading segments... Jul 27 00:07:11.755291858 (60h31m21.477854969s) [upload=58048]: Got next segment. Inline: false Jul 27 00:07:11.755341433 (60h31m21.477904474s) [upload=58048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:11.755345661 (60h31m21.477908701s) [upload=58048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:11.990165825 (60h31m21.712728865s) [seg_pos={0 0}, upload=58048]: Begin upload segment... Jul 27 00:07:11.990211012 (60h31m21.712774052s) [seg_pos={0 0}, upload=58048]: Done begin upload segment. Jul 27 00:07:11.990221621 (60h31m21.712784652s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990230568 (60h31m21.712793608s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990633510 (60h31m21.71319655s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990967869 (60h31m21.713530899s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.991721360 (60h31m21.7142844s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992179327 (60h31m21.714742367s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992587718 (60h31m21.715150759s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992994487 (60h31m21.715557527s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993531444 (60h31m21.716094484s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993948862 (60h31m21.716511902s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.994373135 (60h31m21.716936175s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995537432 (60h31m21.718100472s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995812709 (60h31m21.718375749s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996213716 (60h31m21.718776756s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996482962 (60h31m21.719046002s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996721988 (60h31m21.719285028s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:13.734955056 (60h31m23.457518096s) [upload=58048]: Next returned nil segment Jul 27 00:07:13.734972579 (60h31m23.457535619s) [upload=58048]: Waiting for error group managing segments... Jul 27 00:07:14.013610340 (60h31m23.73617338s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018322977 (60h31m23.740886027s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018799488 (60h31m23.741362528s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.089681915 (60h31m23.812244955s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.098404755 (60h31m23.820967795s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.106109907 (60h31m23.828672947s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.112275553 (60h31m23.834838583s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.113721300 (60h31m23.83628433s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.117463299 (60h31m23.840026339s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.178263677 (60h31m23.900826717s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.464873981 (60h31m24.187437021s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:15.435585766 (60h31m25.158148796s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:41.107251487 (60h31m50.829814527s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107464594 (60h31m50.830027664s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:57142->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107538756 (60h31m50.830101827s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38974->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107575747 (60h31m50.830138787s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:07:41.107609272 (60h31m50.830172302s) [seg_pos={0 0}, upload=58048]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:07:41.340882517 (60h31m51.063445547s) [upload=58048]: Done uploading segments... Jul 27 00:07:41.340925319 (60h31m51.063488369s) [upload=58048]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235823725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXX] duration: 30.241106914s FAIL (2025-07-27 00:07:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.24111975s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX..] duration: 5.897942536s FAIL (2025-07-27 00:07:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.897964458s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.862636221s FAIL (2025-07-27 00:07:23 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.862666029s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231435993s FAIL (2025-07-27 00:07:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== item: pb.BatchRequestItem_SegmentBegin Jul 27 00:06:23.264389462 (60h30m32.986952502s) [seg_pos={0 0}, upload=58035]: Begin upload segment... Jul 27 00:06:23.264439347 (60h30m32.987002387s) [seg_pos={0 0}, upload=58035]: Done begin upload segment. Jul 27 00:06:23.264466770 (60h30m32.98702982s) [noise=true , address=79.127.219.38:4009 , piece_id=UFE4OFSPUN7UGZPMHYGCXVW5FIN7DGYCJYTE7CSP2QVQCI5HI2YQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.264525191 (60h30m32.987088231s) [noise=true , address=104.36.82.47:4552 , piece_id=QXIPZPL2A7KHS6IRFJI7HC47DYLRX2QTYCO5T3PZJCFA2LP44WPQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.264869410 (60h30m32.987432451s) [noise=true , address=104.36.82.46:4497 , piece_id=3YHTH7QRCV3AZZXJQF3NPABNSLZ3FNK3EC73MXAGUCYCZGTXGRGA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.264989820 (60h30m32.987552861s) [noise=true , address=79.127.219.51:4212 , piece_id=RU7AVMQFKGLT4UV7USAWN4UABHH74EDX52URHTEMHGJDAE73H37A, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265183531 (60h30m32.987746571s) [noise=true , address=77.90.31.12:4611 , piece_id=SLA2SPTQIK3ZIVKUZVHMWF26NFIOYOWYOARZRT7TNRVKRUKNYM7Q, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265344980 (60h30m32.98790802s) [noise=true , address=64.49.13.26:4953 , piece_id=ZQM5HCJHVI3JRM76N3KVIDIWVKDXSVUZAXX5LOHX6XW7UD5R4XTQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265523262 (60h30m32.988086302s) [noise=true , address=104.36.82.53:4124 , piece_id=F5HKMKDRGAAUYTVE3T45MXPKKPHZG3J6RI4NQZKARBILA3HSBRAQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265619225 (60h30m32.988182265s) [noise=true , address=104.36.82.54:4169 , piece_id=62OVJCI54IJRCDUJW7H5EMEKM6AOCXQELRSZZ3APAJIMLWGOFOPA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265851629 (60h30m32.988414669s) [noise=true , address=77.90.31.11:4579 , piece_id=ZKEVLZTA62BYPOUR4RZOK236EAH7S5KK3HVRRFTPD7KOGGGKRFBQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266138869 (60h30m32.988701909s) [noise=true , address=79.127.201.215:4359 , piece_id=G72XJNM23VWHUURJ56JSFDR7G3JTBHPRAMUEIFITB62GPBHDYTTQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266410870 (60h30m32.98897391s) [noise=true , address=104.36.82.56:4264 , piece_id=FFWW5EL7S2A7H7D3VCZYBZTIRHGYGKIY2SQQLYVA6NN3XEDG35OQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266707317 (60h30m32.989270357s) [noise=true , address=79.127.205.246:4288 , piece_id=UGXYSTWYQL6INNZXXFN43QV2MQ4F4RYBP7BJ5UNQU3RUY36EDICA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266724219 (60h30m32.989287259s) [noise=true , address=77.90.31.13:4644 , piece_id=KJ4ARAULFCNVJJUH7PDAU54VZWDTQFDEQM2CULAJGBQMKF2MVUAA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266955372 (60h30m32.989518412s) [noise=true , address=79.127.205.234:4614 , piece_id=Y2DYAZYKYVWRWMQMPBHC3VEDQWI6DPXUN4KFCV47VJXX3WCLC7UA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266990118 (60h30m32.989553158s) [noise=true , address=64.49.13.138:4483 , piece_id=M2S7IXL2YCAVIXE3QM3S6YQAC3WCJQSURGFU6EJEXUMERE5JNP4A, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.267183307 (60h30m32.989746348s) [noise=true , address=156.146.43.225:4448 , piece_id=WVEOI6VV55WP5FH7ZMNCRG5P6DSYXQP5UTUOR4M5LEZ37V3MP6AQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.267276024 (60h30m32.989839065s) [noise=true , address=79.127.220.101:4074 , piece_id=FHWYUCW5CPJXAP4OQOQTSHFVA4NIQU7OYMYWHLP5ONBRFBEVI4GQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.267400734 (60h30m32.989963784s) [noise=true , address=77.90.31.15:4713 , piece_id=VVL6ZJFTPAEOSPLTKSGPZXNLEO5N2446KPBROMZ6ODPKFKF46SPQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:24.387683351 (60h30m34.110246391s) [upload=58035]: Next returned nil segment Jul 27 00:06:24.387697508 (60h30m34.110260548s) [upload=58035]: Waiting for error group managing segments... Jul 27 00:06:24.673095542 (60h30m34.395658582s) [noise=true , address=79.127.205.246:4288 , piece_id=UGXYSTWYQL6INNZXXFN43QV2MQ4F4RYBP7BJ5UNQU3RUY36EDICA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:24.707938928 (60h30m34.430502028s) [noise=true , address=79.127.205.234:4614 , piece_id=Y2DYAZYKYVWRWMQMPBHC3VEDQWI6DPXUN4KFCV47VJXX3WCLC7UA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:24.740682445 (60h30m34.463245485s) [noise=true , address=64.49.13.26:4953 , piece_id=ZQM5HCJHVI3JRM76N3KVIDIWVKDXSVUZAXX5LOHX6XW7UD5R4XTQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:25.403201396 (60h30m35.125764436s) [noise=true , address=77.90.31.11:4579 , piece_id=ZKEVLZTA62BYPOUR4RZOK236EAH7S5KK3HVRRFTPD7KOGGGKRFBQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:25.413026002 (60h30m35.135589042s) [noise=true , address=79.127.219.38:4009 , piece_id=UFE4OFSPUN7UGZPMHYGCXVW5FIN7DGYCJYTE7CSP2QVQCI5HI2YQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:27.646840978 (60h30m37.369404018s) [noise=true , address=77.90.31.13:4644 , piece_id=KJ4ARAULFCNVJJUH7PDAU54VZWDTQFDEQM2CULAJGBQMKF2MVUAA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:27.779892114 (60h30m37.502455154s) [noise=true , address=156.146.43.225:4448 , piece_id=WVEOI6VV55WP5FH7ZMNCRG5P6DSYXQP5UTUOR4M5LEZ37V3MP6AQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:28.688597737 (60h30m38.411160777s) [noise=true , address=79.127.219.51:4212 , piece_id=RU7AVMQFKGLT4UV7USAWN4UABHH74EDX52URHTEMHGJDAE73H37A, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:29.067767433 (60h30m38.790330473s) [noise=true , address=79.127.220.101:4074 , piece_id=FHWYUCW5CPJXAP4OQOQTSHFVA4NIQU7OYMYWHLP5ONBRFBEVI4GQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:29.721835031 (60h30m39.444398071s) [noise=true , address=77.90.31.12:4611 , piece_id=SLA2SPTQIK3ZIVKUZVHMWF26NFIOYOWYOARZRT7TNRVKRUKNYM7Q, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:29.930228928 (60h30m39.652791968s) [noise=true , address=64.49.13.138:4483 , piece_id=M2S7IXL2YCAVIXE3QM3S6YQAC3WCJQSURGFU6EJEXUMERE5JNP4A, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:33.298983067 (60h30m43.021546097s) [noise=true , address=79.127.201.215:4359 , piece_id=G72XJNM23VWHUURJ56JSFDR7G3JTBHPRAMUEIFITB62GPBHDYTTQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:07:22.175214675 (60h31m31.897777715s) [noise=true , address=104.36.82.53:4124 , piece_id=F5HKMKDRGAAUYTVE3T45MXPKKPHZG3J6RI4NQZKARBILA3HSBRAQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: rpc: dial tcp 104.36.82.53:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175526061 (60h31m31.898089101s) [noise=true , address=104.36.82.54:4169 , piece_id=62OVJCI54IJRCDUJW7H5EMEKM6AOCXQELRSZZ3APAJIMLWGOFOPA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: write tcp 188.166.225.170:34160->104.36.82.54:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175557291 (60h31m31.898120332s) [noise=true , address=104.36.82.46:4497 , piece_id=3YHTH7QRCV3AZZXJQF3NPABNSLZ3FNK3EC73MXAGUCYCZGTXGRGA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:12Zg4PSYBuLxFvF9oYiuvn65JNbjT5RdagcVAQLp2izvjPe1NEc, address:104.36.82.46:4497): protocol: write tcp 188.166.225.170:55572->104.36.82.46:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175591988 (60h31m31.898155028s) [noise=true , address=104.36.82.56:4264 , piece_id=FFWW5EL7S2A7H7D3VCZYBZTIRHGYGKIY2SQQLYVA6NN3XEDG35OQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: write tcp 188.166.225.170:45042->104.36.82.56:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175607877 (60h31m31.898170918s) [noise=true , address=77.90.31.15:4713 , piece_id=VVL6ZJFTPAEOSPLTKSGPZXNLEO5N2446KPBROMZ6ODPKFKF46SPQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: write tcp 188.166.225.170:44700->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175623498 (60h31m31.898186538s) [noise=true , address=104.36.82.47:4552 , piece_id=QXIPZPL2A7KHS6IRFJI7HC47DYLRX2QTYCO5T3PZJCFA2LP44WPQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: write tcp 188.166.225.170:60470->104.36.82.47:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175647133 (60h31m31.898210163s) [seg_pos={0 0}, upload=58035]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:07:22.404227229 (60h31m32.126790269s) [upload=58035]: Done uploading segments... Jul 27 00:07:22.404245834 (60h31m32.126808864s) [upload=58035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.231446201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX.] duration: 7.576120302s FAIL (2025-07-27 00:07:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.576143827s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXX] duration: 30.247039811s FAIL (2025-07-27 00:07:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.247060992s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX] duration: 30.000991283s FAIL (2025-07-27 00:07:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001000911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.836285196s FAIL (2025-07-27 00:06:51 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.836308362s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..........] duration: 30.79533043s FAIL (2025-07-27 00:06:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.795354036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.X] duration: 30.00191878s FAIL (2025-07-27 00:06:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00194941s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XX] duration: 30.239745298s FAIL (2025-07-27 00:06:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.239761157s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXX] duration: 30.231596387s FAIL (2025-07-27 00:06:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== s (needed at least 13 but got 10) Jul 27 00:05:57.436782393 (60h30m7.159345433s) [upload=58029]: Starting upload Jul 27 00:05:57.436841255 (60h30m7.159404295s) [upload=58029]: Done starting upload Jul 27 00:05:57.436944794 (60h30m7.159507834s) [upload=58029]: Uploading segments... Jul 27 00:05:57.437036960 (60h30m7.1596s) [upload=58029]: Got next segment. Inline: false Jul 27 00:05:57.437078049 (60h30m7.159641089s) [upload=58029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:05:57.437083178 (60h30m7.159646218s) [upload=58029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:05:57.878627803 (60h30m7.601190843s) [seg_pos={0 0}, upload=58029]: Begin upload segment... Jul 27 00:05:57.878668642 (60h30m7.601231682s) [seg_pos={0 0}, upload=58029]: Done begin upload segment. Jul 27 00:05:57.878693149 (60h30m7.601256189s) [noise=true , address=45.91.116.14:4191 , piece_id=XISWZZRL2XMU2IK5CBH7PVCPTTTSBB7VZP6NOCM4FSQAUDWES5DQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.878743705 (60h30m7.601306745s) [noise=true , address=45.91.116.7:4009 , piece_id=Q7MQKNWWMILXHKEOTXQG3IPSPPJPFMZPVIBLWNYSHNJFJUKOOLYA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879047837 (60h30m7.601610877s) [noise=true , address=77.90.31.17:4772 , piece_id=3RWZFAOC7H2IVAJ4ZELCJYDJ656LKP76FNFGIE4DUSVNZVUBJMVQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879220728 (60h30m7.601783768s) [noise=true , address=104.36.82.7:4910 , piece_id=C5LIRU4ITVNP5CAZRXWUQTDJ5EJDX2MPXPMI3YVEBQYJDFYVZE3A, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879296172 (60h30m7.601859212s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879559025 (60h30m7.602122065s) [noise=true , address=45.91.116.5:4299 , piece_id=TNZQNORDMODB4JVAOJNTWFIEDHYDHHMK3HXY3H7KKMM5IHO6YSXA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879806168 (60h30m7.602369208s) [noise=true , address=104.36.82.7:4911 , piece_id=CQRJMWLXF2IZ3ZAZOS27SUO2KXD6ZGGXCPKAGIKFNB556TG37JRQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.880097045 (60h30m7.602660085s) [noise=true , address=104.36.82.8:4971 , piece_id=KCRFRE2KRFTDFPUF2I5TXMP3AN7YVI543NPMGHGSWKGNL25DZY2A, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.880331633 (60h30m7.602894673s) [noise=true , address=45.91.116.9:4262 , piece_id=PNUIEBHBU6HZEJ6EW6S6NRKATQJTML2GJ6IODKV5FPFQ76BEORJA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.880595328 (60h30m7.603158368s) [noise=true , address=77.90.31.18:4807 , piece_id=HETSBBVKXOLAUTOUUIYFVGJIXR4UNDSYOHUKQIFVETKJV2KV4TTA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.880811973 (60h30m7.603375003s) [noise=true , address=104.36.82.12:4150 , piece_id=KTI4HOHOLP5CF3JCPUZYFPSMSASXJ72NPGRF65NZRN55LA4ECNZQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881019630 (60h30m7.60358267s) [noise=true , address=45.91.116.10:4081 , piece_id=H7M6NVHVBY6IBEXNZCE325DMP7SHHXAJ46AWBHE4D3B3ZOPIBIFA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881231736 (60h30m7.603794776s) [noise=true , address=77.90.31.15:4698 , piece_id=RUG7HTLJXQYCBHXMSK2UJH6BA3BXKCMGLAHUAPELUK4MLOLT3GFA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881410217 (60h30m7.603973257s) [noise=true , address=77.90.31.14:4662 , piece_id=ZNBUOT4CM6AUELDCJZULZRMYY3DPKAQGGAWZ4J47HYZSUL7R2YTQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881641800 (60h30m7.60420483s) [noise=true , address=104.36.82.6:4876 , piece_id=TIXMXTTEYPA57JH6J3G2CIED4ZLS2JLG5KMX5BLKU4HWX5WIHUKQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881728897 (60h30m7.604291927s) [noise=true , address=45.91.116.13:4154 , piece_id=YFBPQS3DQAMJYHT5H32JWZUOVA5MIPJGUIGSVNPGXFKWLBYYGOXQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:59.620668084 (60h30m9.343231124s) [upload=58029]: Next returned nil segment Jul 27 00:05:59.620685557 (60h30m9.343248597s) [upload=58029]: Waiting for error group managing segments... Jul 27 00:05:59.897369139 (60h30m9.619932179s) [noise=true , address=45.91.116.13:4154 , piece_id=YFBPQS3DQAMJYHT5H32JWZUOVA5MIPJGUIGSVNPGXFKWLBYYGOXQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:05:59.913229704 (60h30m9.635792744s) [noise=true , address=45.91.116.7:4009 , piece_id=Q7MQKNWWMILXHKEOTXQG3IPSPPJPFMZPVIBLWNYSHNJFJUKOOLYA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:05:59.976379096 (60h30m9.698942136s) [noise=true , address=45.91.116.10:4081 , piece_id=H7M6NVHVBY6IBEXNZCE325DMP7SHHXAJ46AWBHE4D3B3ZOPIBIFA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:05:59.983423965 (60h30m9.705987005s) [noise=true , address=77.90.31.14:4662 , piece_id=ZNBUOT4CM6AUELDCJZULZRMYY3DPKAQGGAWZ4J47HYZSUL7R2YTQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:05:59.998286812 (60h30m9.720849852s) [noise=true , address=45.91.116.5:4299 , piece_id=TNZQNORDMODB4JVAOJNTWFIEDHYDHHMK3HXY3H7KKMM5IHO6YSXA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:06:00.008836206 (60h30m9.731399236s) [noise=true , address=77.90.31.18:4807 , piece_id=HETSBBVKXOLAUTOUUIYFVGJIXR4UNDSYOHUKQIFVETKJV2KV4TTA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:06:00.124184478 (60h30m9.846747508s) [noise=true , address=45.91.116.9:4262 , piece_id=PNUIEBHBU6HZEJ6EW6S6NRKATQJTML2GJ6IODKV5FPFQ76BEORJA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:06:00.210814585 (60h30m9.933377625s) [noise=true , address=45.91.116.14:4191 , piece_id=XISWZZRL2XMU2IK5CBH7PVCPTTTSBB7VZP6NOCM4FSQAUDWES5DQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:06:26.796472853 (60h30m36.519035893s) [noise=true , address=77.90.31.17:4772 , piece_id=3RWZFAOC7H2IVAJ4ZELCJYDJ656LKP76FNFGIE4DUSVNZVUBJMVQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796538539 (60h30m36.519101579s) [noise=true , address=104.36.82.8:4971 , piece_id=KCRFRE2KRFTDFPUF2I5TXMP3AN7YVI543NPMGHGSWKGNL25DZY2A, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796576962 (60h30m36.519140002s) [noise=true , address=104.36.82.7:4911 , piece_id=CQRJMWLXF2IZ3ZAZOS27SUO2KXD6ZGGXCPKAGIKFNB556TG37JRQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:56848->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796622630 (60h30m36.51918567s) [noise=true , address=77.90.31.15:4698 , piece_id=RUG7HTLJXQYCBHXMSK2UJH6BA3BXKCMGLAHUAPELUK4MLOLT3GFA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:45116->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796661484 (60h30m36.519224524s) [noise=true , address=104.36.82.12:4150 , piece_id=KTI4HOHOLP5CF3JCPUZYFPSMSASXJ72NPGRF65NZRN55LA4ECNZQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796689097 (60h30m36.519252137s) [noise=true , address=104.36.82.6:4876 , piece_id=TIXMXTTEYPA57JH6J3G2CIED4ZLS2JLG5KMX5BLKU4HWX5WIHUKQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796710368 (60h30m36.519273408s) [noise=true , address=104.36.82.7:4910 , piece_id=C5LIRU4ITVNP5CAZRXWUQTDJ5EJDX2MPXPMI3YVEBQYJDFYVZE3A, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.231616634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.824645846s FAIL (2025-07-27 00:06:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.82466946s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.246366359s FAIL (2025-07-27 00:06:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== (60h29m32.51017865s) [noise=true , address=45.91.116.14:4206 , piece_id=RB2ZE5QKZY2QTRK3BZ3V6F6AVJISFJRQ4MRYUSN6FMA6JZSU6ESA, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.787943176 (60h29m32.510506217s) [noise=true , address=104.36.82.52:4091 , piece_id=SLFR5XIA3BYL4G5BKBOKCVOICJUGHJXDXYVXUTMKSZ3ACKPPP3YQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788012329 (60h29m32.510575369s) [noise=true , address=104.36.82.19:4408 , piece_id=YLTM6OBYPAQVZR7W6XXTXWZ2KYRF6BRFGA4NSOZE5G26F4X6TAYA, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788241988 (60h29m32.510805028s) [noise=true , address=45.91.116.9:4277 , piece_id=WBXXC2LSJFSLCHHP2YOWNASDKMD7YHOXGLHZPQZ7LA3CTYIKD3NQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788362539 (60h29m32.510925569s) [noise=true , address=79.127.205.246:4258 , piece_id=5FTQPKUVVCBTWWTIDDVF5QHXIWN6N2D5AEY6EQNZVMLQBM3PBRCQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788489312 (60h29m32.511052342s) [noise=true , address=104.36.82.47:4530 , piece_id=HINBU7RQ7ZP2H6Y3IYHYBVTDSJAFMCD26V3EJBBYSTFUCNKFXLVQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788635791 (60h29m32.511198821s) [noise=true , address=79.127.205.234:4607 , piece_id=DLVFNWFBT5VD5K2YEQLHWFXCOZTRYHLVOSOY4XRVNPML5PFXHMDQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788801899 (60h29m32.511364939s) [noise=true , address=104.36.82.46:4511 , piece_id=7ODT6CSGTF57AJORRAKGPKNKQ5NGNUAP75PN3BTLIEFSZ23EFQAQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788915456 (60h29m32.511478496s) [noise=true , address=77.90.31.10:4523 , piece_id=64GDOWJMFH72AGX7RSPKO5F5QUVZGGVFFFKD5JMZTMBY2F7MITYA, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789048411 (60h29m32.511611451s) [noise=true , address=64.49.13.138:4483 , piece_id=X5RAUYAQPVXEP5PPNYOSBBE4Q3SH6ADMSFAQASZ4BN3YUDD6RZ7A, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789151699 (60h29m32.511714739s) [noise=true , address=104.36.82.56:4231 , piece_id=VZXEM3AKXD37SXJDTLSVQXRVKMC2KN22BKKD4JESEP74W4RRIYWQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789284383 (60h29m32.511847423s) [noise=true , address=104.36.82.53:4125 , piece_id=Z2N7PEESJFYSJ2EWAJVD7HN2YNQ6AT5HEGAEHUKYK26QVPW7ACEQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789392951 (60h29m32.511955991s) [noise=true , address=79.127.201.215:4345 , piece_id=OH4NQ72ANUPWVCW6AKEWFPUZFYQNQTQFZF2WCCBCMMYB5W42XDBQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789488373 (60h29m32.512051413s) [noise=true , address=77.90.31.19:4867 , piece_id=4TGQ2CNWNJ6ZBACLLU7HQ44FPK34ERCNQOKA47NYQLRQC53Z5XQA, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789637699 (60h29m32.512200739s) [noise=true , address=79.127.205.235:4657 , piece_id=EJWS4UFR4RRURJXWVXE46SLA5GDH3RFUYZVLFSZKD4BAUVBSRMNQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789684658 (60h29m32.512247688s) [noise=true , address=104.36.82.54:4172 , piece_id=EJNYO5KJ6F3D3JIZQNGN3N7PHL24ABZTYCUNUCZEOUFV6VBX5QBQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789908547 (60h29m32.512471587s) [noise=true , address=79.127.220.101:4041 , piece_id=5TKGOYKYFD37YM2HSOIZ7LJKAQM55DLVW6QAKUBGUJNOZX5RUTPQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:23.902236566 (60h29m33.624799596s) [upload=58019]: Next returned nil segment Jul 27 00:05:23.902253699 (60h29m33.624816739s) [upload=58019]: Waiting for error group managing segments... Jul 27 00:05:24.227516616 (60h29m33.950079656s) [noise=true , address=79.127.205.235:4657 , piece_id=EJWS4UFR4RRURJXWVXE46SLA5GDH3RFUYZVLFSZKD4BAUVBSRMNQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:24.522516541 (60h29m34.245079581s) [noise=true , address=79.127.205.246:4258 , piece_id=5FTQPKUVVCBTWWTIDDVF5QHXIWN6N2D5AEY6EQNZVMLQBM3PBRCQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:24.787777739 (60h29m34.510340779s) [noise=true , address=77.90.31.16:4759 , piece_id=OV6NCZRVNDFWVK35MASHEG2UFGQG33H6I2PTJJFLRFVPMSABD5ZA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:24.939622529 (60h29m34.662185569s) [noise=true , address=79.127.205.234:4607 , piece_id=DLVFNWFBT5VD5K2YEQLHWFXCOZTRYHLVOSOY4XRVNPML5PFXHMDQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:27.214858358 (60h29m36.937421398s) [noise=true , address=79.127.220.101:4041 , piece_id=5TKGOYKYFD37YM2HSOIZ7LJKAQM55DLVW6QAKUBGUJNOZX5RUTPQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:28.083413763 (60h29m37.805976803s) [noise=true , address=64.49.13.138:4483 , piece_id=X5RAUYAQPVXEP5PPNYOSBBE4Q3SH6ADMSFAQASZ4BN3YUDD6RZ7A, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:29.775391290 (60h29m39.49795432s) [noise=true , address=45.91.116.9:4277 , piece_id=WBXXC2LSJFSLCHHP2YOWNASDKMD7YHOXGLHZPQZ7LA3CTYIKD3NQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:05:31.705915531 (60h29m41.428478571s) [noise=true , address=77.90.31.19:4867 , piece_id=4TGQ2CNWNJ6ZBACLLU7HQ44FPK34ERCNQOKA47NYQLRQC53Z5XQA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:34.195265036 (60h29m43.917828076s) [noise=true , address=45.91.116.14:4206 , piece_id=RB2ZE5QKZY2QTRK3BZ3V6F6AVJISFJRQ4MRYUSN6FMA6JZSU6ESA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:56.693782311 (60h30m6.416345351s) [noise=true , address=79.127.201.215:4345 , piece_id=OH4NQ72ANUPWVCW6AKEWFPUZFYQNQTQFZF2WCCBCMMYB5W42XDBQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:06:21.700033712 (60h30m31.422596782s) [noise=true , address=104.36.82.52:4091 , piece_id=SLFR5XIA3BYL4G5BKBOKCVOICJUGHJXDXYVXUTMKSZ3ACKPPP3YQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.700386977 (60h30m31.422950037s) [noise=true , address=77.90.31.10:4523 , piece_id=64GDOWJMFH72AGX7RSPKO5F5QUVZGGVFFFKD5JMZTMBY2F7MITYA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: read tcp 188.166.225.170:43628->77.90.31.10:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.700918505 (60h30m31.423481545s) [noise=true , address=104.36.82.46:4511 , piece_id=7ODT6CSGTF57AJORRAKGPKNKQ5NGNUAP75PN3BTLIEFSZ23EFQAQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: rpc: dial tcp 104.36.82.46:4511: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.700947921 (60h30m31.423510961s) [noise=true , address=104.36.82.56:4231 , piece_id=VZXEM3AKXD37SXJDTLSVQXRVKMC2KN22BKKD4JESEP74W4RRIYWQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701018105 (60h30m31.423581165s) [noise=true , address=104.36.82.54:4172 , piece_id=EJNYO5KJ6F3D3JIZQNGN3N7PHL24ABZTYCUNUCZEOUFV6VBX5QBQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: write tcp 188.166.225.170:33396->104.36.82.54:4172: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701057230 (60h30m31.42362029s) [noise=true , address=104.36.82.19:4408 , piece_id=YLTM6OBYPAQVZR7W6XXTXWZ2KYRF6BRFGA4NSOZE5G26F4X6TAYA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ, address:104.36.82.19:4408): protocol: rpc: dial tcp 104.36.82.19:4408: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701066748 (60h30m31.423629788s) [noise=true , address=104.36.82.53:4125 , piece_id=Z2N7PEESJFYSJ2EWAJVD7HN2YNQ6AT5HEGAEHUKYK26QVPW7ACEQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:104.36.82.53:4125): protocol: rpc: dial tcp 104.36.82.53:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701071377 (60h30m31.423634417s) [noise=true , address=104.36.82.47:4530 , piece_id=HINBU7RQ7ZP2H6Y3IYHYBVTDSJAFMCD26V3EJBBYSTFUCNKFXLVQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701094822 (60h30m31.423657922s) [seg_pos={0 0}, upload=58019]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:06:21.944972146 (60h30m31.667535186s) [upload=58019]: Done uploading segments... Jul 27 00:06:21.944996322 (60h30m31.667559362s) [upload=58019]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.246385647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.....XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.........] duration: 30.384339509s FAIL (2025-07-27 00:06:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.38435079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.X] duration: 30.23505612s FAIL (2025-07-27 00:05:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235082952s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)