Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.....X.X...........X...................X.........] duration: 5.824073222s FAIL (2025-07-28 00:59:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.82410349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.808394094s FAIL (2025-07-28 00:59:19 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.808420944s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [.......X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23706941s FAIL (2025-07-28 00:59:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 4252 , piece_id=5VSIPIPBFNWLCKJFLAMOVYVSUYB6BTXFSXCW62RULMRE7WFFKAOQ, seg_pos={0 0}, upload=81908]: Uploading piece... Jul 28 00:58:04.832982945 (85h22m14.555545985s) [noise=true , address=104.36.82.11:4145 , piece_id=3XMARM3K237IGAIRKSFN6GLOIMYK334NYKQBFSDOWD7WNKI6UMZQ, seg_pos={0 0}, upload=81908]: Uploading piece... Jul 28 00:58:04.833152479 (85h22m14.555715519s) [noise=true , address=104.36.82.15:4264 , piece_id=7B2JJ4JU43TVFZ6B6CVCC3NGLDKS4RGBVKHA67DTOAERNWY4CUNQ, seg_pos={0 0}, upload=81908]: Uploading piece... Jul 28 00:58:04.833472432 (85h22m14.556035472s) [noise=true , address=104.36.82.22:4539 , piece_id=F3SQHQEMVOHEHLX2SONXVN4OBCSLX6YTA23M6TJBKO6LEZHTQSYA, seg_pos={0 0}, upload=81908]: Uploading piece... Jul 28 00:58:04.833846968 (85h22m14.556410008s) [noise=true , address=104.36.82.53:4149 , piece_id=EKJOPZFBVTPTLGRNBPG6635PVKGV2NR4QT4CMOV6ET54PC7NDNKQ, seg_pos={0 0}, upload=81908]: Uploading piece... Jul 28 00:58:04.834065005 (85h22m14.556628045s) [noise=true , address=104.36.82.42:4822 , piece_id=PEXD5IT3FBPCUKHPYYYCMJMTIH5BZZGEHYIHMJGUXY5MR36QRHAQ, seg_pos={0 0}, upload=81908]: Uploading piece... Jul 28 00:58:04.834337166 (85h22m14.556900206s) [noise=true , address=104.36.82.44:4884 , piece_id=W35VM6MSUOMHON2K7SBWLFDQFM6DW7JZ732KZ7JPJ3MZSZFUKGIA, seg_pos={0 0}, upload=81908]: Uploading piece... Jul 28 00:58:05.964164989 (85h22m15.686728029s) [upload=81908]: Next returned nil segment Jul 28 00:58:05.964180608 (85h22m15.686743648s) [upload=81908]: Waiting for error group managing segments... Jul 28 00:58:06.284468891 (85h22m16.007031931s) [noise=true , address=79.127.205.245:4252 , piece_id=5VSIPIPBFNWLCKJFLAMOVYVSUYB6BTXFSXCW62RULMRE7WFFKAOQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: <nil> Jul 28 00:58:06.733705929 (85h22m16.456268969s) [noise=true , address=79.127.201.214:4305 , piece_id=SW2USWHSCOTRZQSY7BML5THL5V3HXR7QNFVXK2X25IFW7EQHWVSQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: <nil> Jul 28 00:59:03.724892912 (85h23m13.447455952s) [noise=true , address=104.36.82.44:4884 , piece_id=W35VM6MSUOMHON2K7SBWLFDQFM6DW7JZ732KZ7JPJ3MZSZFUKGIA, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: rpc: dial tcp 104.36.82.44:4884: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.724943728 (85h23m13.447506768s) [noise=true , address=104.36.82.9:4008 , piece_id=M3C64IU5LYU53DVV5XZBOVEHZHQOTTY7TDEA7J3YQEV5O7TMNDLQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.724975370 (85h23m13.44753841s) [noise=true , address=104.36.82.6:4894 , piece_id=S4NHJ24OJSS5AZPBVDPQH2MXB4ZUBOJB2DY3SPGKSOLDWWNJMGSQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:12fZsj77XZ8CnYD6iwzEVKPn6yLyxWMtibzsRXcuGKwNxEXB6Hq, address:104.36.82.6:4894): protocol: write tcp 188.166.225.170:39246->104.36.82.6:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725286595 (85h23m13.447849635s) [noise=true , address=104.36.82.10:4098 , piece_id=27VF2AIFKO3XNA4PLMCHPUGV6LLDG72E3TFNXTB4POTBDECCYBYQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:55396->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725340999 (85h23m13.447904039s) [noise=true , address=104.36.82.55:4220 , piece_id=LJ6YP3OHSFI6UO7VK4NW4JQARF5ZCLN7RJVVUWJHE5ULCT5ZZXMA, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 188.166.225.170:59040->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725373411 (85h23m13.447936451s) [noise=true , address=104.36.82.53:4149 , piece_id=EKJOPZFBVTPTLGRNBPG6635PVKGV2NR4QT4CMOV6ET54PC7NDNKQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: write tcp 188.166.225.170:55640->104.36.82.53:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725419830 (85h23m13.44798287s) [noise=true , address=104.36.82.52:4088 , piece_id=CVDK4BKLJQUAJNLYNFZDOFRS3AH553ZVI362YAFV4CNFUBONDGPQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:104.36.82.52:4088): protocol: write tcp 188.166.225.170:46834->104.36.82.52:4088: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725484153 (85h23m13.448047193s) [noise=true , address=104.36.82.42:4822 , piece_id=PEXD5IT3FBPCUKHPYYYCMJMTIH5BZZGEHYIHMJGUXY5MR36QRHAQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:104.36.82.42:4822): protocol: write tcp 188.166.225.170:37212->104.36.82.42:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725540761 (85h23m13.448103801s) [noise=true , address=104.36.82.49:4980 , piece_id=GKVHAQDG67BYDUIBBFSN5NEJ6UNS2HYIBLXETWS76WUSLC5MSHOA, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1hzDqFsTcWDF5LLW9tm77hLADjUkhbLqkE3DBSHyJf3PyiHszu, address:104.36.82.49:4980): protocol: write tcp 188.166.225.170:41662->104.36.82.49:4980: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725571019 (85h23m13.448134059s) [noise=true , address=104.36.82.11:4145 , piece_id=3XMARM3K237IGAIRKSFN6GLOIMYK334NYKQBFSDOWD7WNKI6UMZQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 188.166.225.170:36098->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725577040 (85h23m13.44814008s) [noise=true , address=104.36.82.22:4539 , piece_id=F3SQHQEMVOHEHLX2SONXVN4OBCSLX6YTA23M6TJBKO6LEZHTQSYA, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:4539): protocol: write tcp 188.166.225.170:41474->104.36.82.22:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725601657 (85h23m13.448164697s) [noise=true , address=104.36.82.15:4264 , piece_id=7B2JJ4JU43TVFZ6B6CVCC3NGLDKS4RGBVKHA67DTOAERNWY4CUNQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1ovYsPwYdx7AYPboFx3Xdcg4VQrPGZj2XTMTqgox6zm81gbf8Z, address:104.36.82.15:4264): protocol: write tcp 188.166.225.170:34638->104.36.82.15:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725612689 (85h23m13.448175729s) [noise=true , address=104.36.82.13:4190 , piece_id=BDMGI2RZ2QSULRHAWZZWFK7ZEKBQUUIAX3LZV5NX6NMBVZSXGUSA, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1eNJzs5Dc7Nv7imoQXZDp2mEVWjD8o4viGVkFvuAeZ1arynxu4, address:104.36.82.13:4190): protocol: write tcp 188.166.225.170:46902->104.36.82.13:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725632226 (85h23m13.448195266s) [noise=true , address=104.36.82.19:4415 , piece_id=X7XK33UWUDMRAPIXQ23YNLR6YVCZV6ZG3VKQXQDJJSSM3UX7NMXA, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1NowH5f2M56Exu7yWkad5uP9tngmKoVsJvmT2WpSkP2PBUi322, address:104.36.82.19:4415): protocol: write tcp 188.166.225.170:37896->104.36.82.19:4415: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725665991 (85h23m13.448229031s) [noise=true , address=104.36.82.54:4177 , piece_id=UVUGXCJRL42L6OSAXG4VEHMTFA6DT4JHWKVKYOCN6OFGGNHUYL7Q, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 188.166.225.170:60904->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725670109 (85h23m13.448233139s) [noise=true , address=104.36.82.50:4034 , piece_id=HIT7236363COA2XQNWSOFZOT2B3BFNHXDTA4YKM7UIX7KARPOFWQ, seg_pos={0 0}, upload=81908]: Done uploading piece. err: ecclient: upload failed (node:12q2ocJMc2bzMsrFH4heg52ZBziCHNxQc7cxDGU79AeUZxqoB9r, address:104.36.82.50:4034): protocol: write tcp 188.166.225.170:34834->104.36.82.50:4034: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:59:03.725689085 (85h23m13.448252125s) [seg_pos={0 0}, upload=81908]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:59:03.960218196 (85h23m13.682781236s) [upload=81908]: Done uploading segments... Jul 28 00:59:03.960242754 (85h23m13.682805794s) [upload=81908]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.237082185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.XX....X....X.......X.....X...........................] duration: 5.965392075s FAIL (2025-07-28 00:59:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.965417764s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..........X.....X.X...........X...................X........] duration: 5.831183257s FAIL (2025-07-28 00:58:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.831225568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.82681574s FAIL (2025-07-28 00:58:30 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.82684236s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.XX....X....X.......X.....X..........................] duration: 5.740191863s FAIL (2025-07-28 00:58:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.740214537s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [........X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231406887s FAIL (2025-07-28 00:58:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 977090 (85h21m14.08154013s) [noise=true , address=104.36.82.42:4809 , piece_id=QINPXJC7EGS2W3DKZC42KKVQDW3OYD3Z2XBVBYZHFUDREPCYH32A, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:04.359141715 (85h21m14.081704755s) [noise=true , address=104.36.82.55:4194 , piece_id=GQX3Q3FXNOS6HHNIFSRGMVEA75MWKG6B5EYSKRTIRRVDXGGMQY5A, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:04.359208662 (85h21m14.081771702s) [noise=true , address=79.127.201.215:4354 , piece_id=ET4CR5YRZNCXOA535AAQ2Z2L6JTZDL4VTPYVAQGP5ZVCC4DMSNHA, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:04.359349642 (85h21m14.081912682s) [noise=true , address=104.36.82.54:4181 , piece_id=BDWJYBKOMI5MRMZDD7Y3IODTBAJC55J2WMWB46TFYPDIZMBBZINQ, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:04.359442641 (85h21m14.082005681s) [noise=true , address=104.36.82.13:4218 , piece_id=XL25ALLPPCMQEVP7A3GSGCF63FCXPWZPWU4BDHHHM5X73IESBWDQ, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:04.359562880 (85h21m14.08212592s) [noise=true , address=104.36.82.5:4844 , piece_id=IOAUEI7NBPARSTT3SFVV2RGH57FN7EQYZMM4JXIBVP3W7JNVLHGA, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:04.359765709 (85h21m14.082328749s) [noise=true , address=104.36.82.41:4764 , piece_id=7QKVLNK7RZHMDNUG6PVYOC5MPSWAPPAD6CV42VJ34RNDFKS4YRZA, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:04.359773163 (85h21m14.082336193s) [noise=true , address=104.36.82.14:4227 , piece_id=EBJB4SVM2CVXA62VUQFI2YA573CVLMFQYLAJWQCLDF6OZQS253FQ, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:04.360879289 (85h21m14.083442329s) [noise=true , address=79.127.201.215:4360 , piece_id=EY62O77LBOIJU4OBTLLGIP2AFVY66RPYNEHORVVRX3MZRB2LXJKA, seg_pos={0 0}, upload=81892]: Uploading piece... Jul 28 00:57:05.861599918 (85h21m15.584162958s) [upload=81892]: Next returned nil segment Jul 28 00:57:05.861615307 (85h21m15.584178347s) [upload=81892]: Waiting for error group managing segments... Jul 28 00:57:07.807302464 (85h21m17.529865504s) [noise=true , address=79.127.201.215:4354 , piece_id=ET4CR5YRZNCXOA535AAQ2Z2L6JTZDL4VTPYVAQGP5ZVCC4DMSNHA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil> Jul 28 00:57:08.501868246 (85h21m18.224431286s) [noise=true , address=79.127.201.215:4360 , piece_id=EY62O77LBOIJU4OBTLLGIP2AFVY66RPYNEHORVVRX3MZRB2LXJKA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil> Jul 28 00:58:03.271178669 (85h22m12.993741709s) [noise=true , address=104.36.82.11:4136 , piece_id=QRSNSY47WFEC4J3IFZSWYJY5GNWWCT5IVPR2AX33G5NXN4FZZERA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:127n8CMbZQ7haiv6tGHrVg9V8Yf12sYkcbL9mEbSpQ4WkT9isnp, address:104.36.82.11:4136): protocol: rpc: dial tcp 104.36.82.11:4136: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271222814 (85h22m12.993785854s) [noise=true , address=104.36.82.54:4181 , piece_id=BDWJYBKOMI5MRMZDD7Y3IODTBAJC55J2WMWB46TFYPDIZMBBZINQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gGf5E7tzF2TXtLWgpK55t24SC1BpTSeC9G66nPmR1ARnDrAE8, address:104.36.82.54:4181): protocol: rpc: dial tcp 104.36.82.54:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271252019 (85h22m12.993815059s) [noise=true , address=104.36.82.41:4764 , piece_id=7QKVLNK7RZHMDNUG6PVYOC5MPSWAPPAD6CV42VJ34RNDFKS4YRZA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271281415 (85h22m12.993844456s) [noise=true , address=104.36.82.47:4539 , piece_id=B7VK5XTLW6LTCY5GXCFDF4XGTNVX2DKVNO5JSECRISFMG75PVJBQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: rpc: dial tcp 104.36.82.47:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271308708 (85h22m12.993871748s) [noise=true , address=104.36.82.20:4468 , piece_id=X6KPCZWKOSRUSUYQTSACWLUGVGH22ZYDXEKKL7WOF2JEDYVRSZNA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:4468): protocol: rpc: dial tcp 104.36.82.20:4468: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271343554 (85h22m12.993906594s) [noise=true , address=104.36.82.51:4069 , piece_id=EXDWE5LC6F4OO7QKT3L33ZBLDH2ACCPNVIEP4OAYAGSDIVGD5NAA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe, address:104.36.82.51:4069): protocol: write tcp 188.166.225.170:32980->104.36.82.51:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271360697 (85h22m12.993923737s) [noise=true , address=104.36.82.56:4242 , piece_id=PXDMODG5DCXVJDXPQWFGDQSXDUXFOEGBXP2DHR44O5CS4U5MLWNA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: rpc: dial tcp 104.36.82.56:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271397838 (85h22m12.993960878s) [noise=true , address=104.36.82.45:4924 , piece_id=SAILGFF46ETZTBZSEZPEAMWNKSY7PZ5QKKIJGNNDLZ2E25U4WMVQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:104.36.82.45:4924): protocol: rpc: dial tcp 104.36.82.45:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271408329 (85h22m12.993971369s) [noise=true , address=104.36.82.13:4218 , piece_id=XL25ALLPPCMQEVP7A3GSGCF63FCXPWZPWU4BDHHHM5X73IESBWDQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc, address:104.36.82.13:4218): protocol: rpc: dial tcp 104.36.82.13:4218: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271434658 (85h22m12.993997698s) [noise=true , address=104.36.82.14:4227 , piece_id=EBJB4SVM2CVXA62VUQFI2YA573CVLMFQYLAJWQCLDF6OZQS253FQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:104.36.82.14:4227): protocol: rpc: dial tcp 104.36.82.14:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271440400 (85h22m12.99400344s) [noise=true , address=104.36.82.44:4889 , piece_id=XN55LTYGRNWDPQ3JJSUPKSZ3UE6WW4T2HG46U6AVW6ZO5Z23PJ4A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1PCtJRFHPHi1dHJNcRnGZg8HZPQDfo5L3aw6DgVnUzVUz92Kyv, address:104.36.82.44:4889): protocol: rpc: dial tcp 104.36.82.44:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271469536 (85h22m12.994032576s) [noise=true , address=104.36.82.12:4165 , piece_id=5DMUM7KRKX7SJSUUD6CVUMWALLWKKUXZH5OETTV3L3JDARXUEUFA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:13Ksy7YCGV5u3T5Z6WwKtMjFq6Rs4bayzR7mQFyXbSFuJjaH4F, address:104.36.82.12:4165): protocol: rpc: dial tcp 104.36.82.12:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271478973 (85h22m12.994042013s) [noise=true , address=104.36.82.10:4106 , piece_id=JVHBXUSMDYYZOT5JQCA2TU7ATBPT3FSCD47CWA2L5RAHUXDLGXYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12CsQq2BuV9hNc5fVAsw882NkDJMEXZeNwhV2FGNGKAU6wuFBHm, address:104.36.82.10:4106): protocol: rpc: dial tcp 104.36.82.10:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271492780 (85h22m12.99405582s) [noise=true , address=104.36.82.5:4844 , piece_id=IOAUEI7NBPARSTT3SFVV2RGH57FN7EQYZMM4JXIBVP3W7JNVLHGA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 188.166.225.170:40386->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271506847 (85h22m12.994069887s) [noise=true , address=104.36.82.55:4194 , piece_id=GQX3Q3FXNOS6HHNIFSRGMVEA75MWKG6B5EYSKRTIRRVDXGGMQY5A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: rpc: dial tcp 104.36.82.55:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271510854 (85h22m12.994073894s) [noise=true , address=104.36.82.42:4809 , piece_id=QINPXJC7EGS2W3DKZC42KKVQDW3OYD3Z2XBVBYZHFUDREPCYH32A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1VZgucaa2gRuFhNxARMb7BDS6EGZXKChGPppbFksbPc5we7fTQ, address:104.36.82.42:4809): protocol: rpc: dial tcp 104.36.82.42:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:58:03.271531644 (85h22m12.994094674s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:58:03.501202360 (85h22m13.2237654s) [upload=81892]: Done uploading segments... Jul 28 00:58:03.501230464 (85h22m13.223793504s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.231420845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..........X.....X.X...........X...................X.......] duration: 5.970126051s FAIL (2025-07-28 00:57:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.970151199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X.XX....X....X.......X.....X.........................] duration: 5.8578957s FAIL (2025-07-28 00:57:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.857919285s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.......X.XX....X....X.......X.....X........................] duration: 5.824735643s FAIL (2025-07-28 00:57:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.824762745s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.86000506s FAIL (2025-07-28 00:57:26 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.860033724s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X..........X.....X.X...........X...................X......] duration: 5.825508694s FAIL (2025-07-28 00:57:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.825531669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [.........X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.247748225s FAIL (2025-07-28 00:57:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== m13.606713719s) [noise=true , address=104.36.82.19:4422 , piece_id=K4IDVIGGBEQGCSS37A2E6ISQCHE5IIPRFXLFCJCM2WU33AVTWBDQ, seg_pos={0 0}, upload=81876]: Uploading piece... Jul 28 00:56:03.884545104 (85h20m13.607108144s) [noise=true , address=104.36.82.3:4793 , piece_id=UEMA7ZBYDBR5SN6KEIAG6XHQWJJYFXQK4PAHQW6FQZH4RBZLGKZA, seg_pos={0 0}, upload=81876]: Uploading piece... Jul 28 00:56:03.884795923 (85h20m13.607358964s) [noise=true , address=45.91.116.9:4277 , piece_id=EZOINCD6GA7EBV7O6QHSHYHWXEU2HEV7SJP7FSUJKGML7SPODO2A, seg_pos={0 0}, upload=81876]: Uploading piece... Jul 28 00:56:03.885005254 (85h20m13.607568294s) [noise=true , address=104.36.82.15:4285 , piece_id=RFWCA53EMJLQLPENZB4PD64JETG2Z4SJ52LKQSOTHUVJXHYS6IDQ, seg_pos={0 0}, upload=81876]: Uploading piece... Jul 28 00:56:03.885207251 (85h20m13.607770291s) [noise=true , address=104.36.82.13:4197 , piece_id=AFBGTRRP7Z3XHBHR7WY757KXSPIH37OSPB7QBPC22PSNV4B4EDMA, seg_pos={0 0}, upload=81876]: Uploading piece... Jul 28 00:56:03.885408876 (85h20m13.607971916s) [noise=true , address=104.36.82.10:4100 , piece_id=X4JJG5DHFLHF36SD4CALR7LT6YRCHE55RCXBDHRFKG4YVNFYDOZQ, seg_pos={0 0}, upload=81876]: Uploading piece... Jul 28 00:56:03.885611685 (85h20m13.608174725s) [noise=true , address=104.36.82.53:4145 , piece_id=FU443QBXAFX3R7ZRMQNWE4NETY5CVSVMQCYXDQODLQ7SIVQUMYJQ, seg_pos={0 0}, upload=81876]: Uploading piece... Jul 28 00:56:03.885859469 (85h20m13.608422509s) [noise=true , address=104.36.82.21:4490 , piece_id=NZPDOAWMFZTL2JWPVBPDRJLZEGQXRARDF3T5NR7YVZ6TNPXQ2WEQ, seg_pos={0 0}, upload=81876]: Uploading piece... Jul 28 00:56:05.516336637 (85h20m15.238899677s) [upload=81876]: Next returned nil segment Jul 28 00:56:05.516355534 (85h20m15.238918584s) [upload=81876]: Waiting for error group managing segments... Jul 28 00:56:06.147318336 (85h20m15.869881376s) [noise=true , address=45.91.116.9:4277 , piece_id=EZOINCD6GA7EBV7O6QHSHYHWXEU2HEV7SJP7FSUJKGML7SPODO2A, seg_pos={0 0}, upload=81876]: Done uploading piece. err: <nil> Jul 28 00:57:02.783734018 (85h21m12.506297058s) [noise=true , address=104.36.82.10:4100 , piece_id=X4JJG5DHFLHF36SD4CALR7LT6YRCHE55RCXBDHRFKG4YVNFYDOZQ, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:12Hyc6sbtuPXMLyFzubMcepbJyLyVAGNpkXAmqg73i3sY97dwdW, address:104.36.82.10:4100): protocol: rpc: dial tcp 104.36.82.10:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.783783472 (85h21m12.506346513s) [noise=true , address=104.36.82.21:4490 , piece_id=NZPDOAWMFZTL2JWPVBPDRJLZEGQXRARDF3T5NR7YVZ6TNPXQ2WEQ, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n, address:104.36.82.21:4490): protocol: rpc: dial tcp 104.36.82.21:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.783812668 (85h21m12.506375708s) [noise=true , address=104.36.82.22:4531 , piece_id=DLKPXMH3GRNGKSYHJANK74IA67ZNTOWAOPTMGLRSGCIJOCDBSAYA, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:12jGenn3Fh2VB36jVPzyy2LxJgbLraXxL2yobw9tsBJLSDPJycN, address:104.36.82.22:4531): protocol: rpc: dial tcp 104.36.82.22:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.783840532 (85h21m12.506403572s) [noise=true , address=104.36.82.44:4884 , piece_id=45VDYQ3RZDXPGJGCQQ7QVFAHJKW3SNPD7FFEOILAH27MCTFMU32A, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: rpc: dial tcp 104.36.82.44:4884: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.783872322 (85h21m12.506435362s) [noise=true , address=104.36.82.52:4095 , piece_id=CRSIBFBRZUAAMB3NMNKGOFWCHKAXVNJJKUY4LNNHTWYJJYA27FJA, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: rpc: dial tcp 104.36.82.52:4095: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.783902971 (85h21m12.506466021s) [noise=true , address=104.36.82.56:4233 , piece_id=KREJNOWOVLL3SLZOZKGRVIVKIMCCFYGPR7KGX42G6BDZLE3B7JNA, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.783937447 (85h21m12.506500488s) [noise=true , address=104.36.82.19:4422 , piece_id=K4IDVIGGBEQGCSS37A2E6ISQCHE5IIPRFXLFCJCM2WU33AVTWBDQ, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1k7Bs5nYevGzsVWhp85cgDsbrpzbefborGwMAAnDsbeT9x7gio, address:104.36.82.19:4422): protocol: rpc: dial tcp 104.36.82.19:4422: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.783966253 (85h21m12.506529293s) [noise=true , address=104.36.82.3:4793 , piece_id=UEMA7ZBYDBR5SN6KEIAG6XHQWJJYFXQK4PAHQW6FQZH4RBZLGKZA, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:104.36.82.3:4793): protocol: rpc: dial tcp 104.36.82.3:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.783999646 (85h21m12.506562686s) [noise=true , address=104.36.82.53:4145 , piece_id=FU443QBXAFX3R7ZRMQNWE4NETY5CVSVMQCYXDQODLQ7SIVQUMYJQ, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: rpc: dial tcp 104.36.82.53:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784030675 (85h21m12.506593716s) [noise=true , address=104.36.82.14:4239 , piece_id=2BULYKVHNEHNCS364NJ52JFOJ542AFA4J2KDBJ6HYCOX3CHHC3PA, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:104.36.82.14:4239): protocol: rpc: dial tcp 104.36.82.14:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784063568 (85h21m12.506626608s) [noise=true , address=104.36.82.12:4172 , piece_id=UMUVUP7M7LJTU2RJPQTLZP5ICV3QO4VOSYQTSO6LVAXLCCBUBUQQ, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1xfh8U6X8B8bpXvyX7JHtAGY6zsApfjQtZghScuo7BerzDj4yj, address:104.36.82.12:4172): protocol: rpc: dial tcp 104.36.82.12:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784092463 (85h21m12.506655503s) [noise=true , address=104.36.82.15:4285 , piece_id=RFWCA53EMJLQLPENZB4PD64JETG2Z4SJ52LKQSOTHUVJXHYS6IDQ, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y, address:104.36.82.15:4285): protocol: rpc: dial tcp 104.36.82.15:4285: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784111440 (85h21m12.50667448s) [noise=true , address=104.36.82.54:4188 , piece_id=UXZTA4AWNCBMANERBSFQDTIG2AJLRRYGHESJAUBKSKOQRGUJR6EA, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG, address:104.36.82.54:4188): protocol: rpc: dial tcp 104.36.82.54:4188: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784116109 (85h21m12.506679169s) [noise=true , address=104.36.82.16:4315 , piece_id=KRURTG256PF7TPAAS7YMAT7WOGOW4S4UEGGDN4F4DSVENE27WQIA, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac, address:104.36.82.16:4315): protocol: rpc: dial tcp 104.36.82.16:4315: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784139353 (85h21m12.506702393s) [noise=true , address=104.36.82.8:4996 , piece_id=CWR3LMB7UPSVJDOH7U7E2QJ72HQ3CNHHJQEU2CDKEO6HKXAH4VYQ, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:12r6XcE2fb6YxU6kez4UMPeq82jNPi7YiRNmi33tQcTVYMYrMV4, address:104.36.82.8:4996): protocol: rpc: dial tcp 104.36.82.8:4996: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784176915 (85h21m12.506739965s) [noise=true , address=104.36.82.5:4865 , piece_id=SUL5B6Q4SLC3R4LLU34VKUTSLCFJAUK4SQXME5ABVXDV24PA6CPQ, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1kqNGahsBw6VsV3JTALvPyGshuSFaAdRSo94cQnCXF3ak54KtL, address:104.36.82.5:4865): protocol: write tcp 188.166.225.170:48920->104.36.82.5:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784182456 (85h21m12.506745496s) [noise=true , address=104.36.82.13:4197 , piece_id=AFBGTRRP7Z3XHBHR7WY757KXSPIH37OSPB7QBPC22PSNV4B4EDMA, seg_pos={0 0}, upload=81876]: Done uploading piece. err: ecclient: upload failed (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM, address:104.36.82.13:4197): protocol: write tcp 188.166.225.170:40532->104.36.82.13:4197: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:57:02.784196142 (85h21m12.506759182s) [seg_pos={0 0}, upload=81876]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:57:03.029376141 (85h21m12.751939241s) [upload=81876]: Done uploading segments... Jul 28 00:57:03.029407972 (85h21m12.751971062s) [upload=81876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.247771639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.X..........X.....X.X...........X...................X.....] duration: 5.829506644s FAIL (2025-07-28 00:56:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.829538295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.869482697s FAIL (2025-07-28 00:56:34 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.869506181s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.......X.XX....X....X.......X.....X.......................] duration: 5.724413179s FAIL (2025-07-28 00:56:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.724443327s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X.......................X...................................] duration: 1.847245991s FAIL (2025-07-28 00:56:15 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.847280548s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.......X.XX....X....X.......X.....X......................] duration: 5.889048925s FAIL (2025-07-28 00:56:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.889075626s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [..........X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233128412s FAIL (2025-07-28 00:55:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== WAXZCC3LGDGUXDB2DKXC3NSVX3427PV3VKMVT36DUHUQ, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:58.070093303 (85h19m7.792656343s) [noise=true , address=104.36.82.56:4233 , piece_id=OADJE5KYPFDQAZGA6HYMKD2FTGTUOLOHSDMUS7NSOSN5SXHQYLVQ, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:58.070224054 (85h19m7.792787094s) [noise=true , address=104.36.82.4:4813 , piece_id=EVVOMMGNKU6SEIUGXB2J4RWGPTJGQ6TKL7GN7FRRWKRQJXUCTBDA, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:58.070361116 (85h19m7.792924156s) [noise=true , address=104.36.82.47:4524 , piece_id=2LGWQ56OHU6IMV3IJQGVKNM4BE66MYTHELVQCW2GRAJXHX3GINOA, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:58.070440608 (85h19m7.793003648s) [noise=true , address=104.36.82.54:4185 , piece_id=LTPYIGXFB3SGAJF2YA7ZALRIPTJ3OLZL22ZWIW6DP7D6OHHDHCYA, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:58.070699002 (85h19m7.793262032s) [noise=true , address=104.36.82.53:4128 , piece_id=WABIAO3HBNS4NYPDI2S2JQILBR66ISPJ5URZ7XWKRXJ2XITH4EKQ, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:58.070804644 (85h19m7.793367704s) [noise=true , address=104.36.82.14:4236 , piece_id=VLT4LD7V3DCPZMHV4ZC6S2KABEGQBZBITFUVMY4XCXJLVJIIYR4A, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:58.071216272 (85h19m7.793779312s) [noise=true , address=104.36.82.11:4138 , piece_id=V7YI2ECXX67FILI74NM7LTK5257L7Q573CJHEW5MHM4WSWK5PEFQ, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:58.071473764 (85h19m7.794036804s) [noise=true , address=104.36.82.12:4158 , piece_id=XUWVKRTXCMDW5JM7BBQIY4EV3YY5MMDOTX6NCNUN4UM2ZJXN4BYA, seg_pos={0 0}, upload=81860]: Uploading piece... Jul 28 00:54:59.612903460 (85h19m9.33546649s) [upload=81860]: Next returned nil segment Jul 28 00:54:59.612921044 (85h19m9.335484074s) [upload=81860]: Waiting for error group managing segments... Jul 28 00:55:05.060779377 (85h19m14.783342437s) [noise=true , address=79.127.201.215:4355 , piece_id=A43APM4U5B2ZACPVXA2DRSKD26R6G6DFBK7UFOW65LE7THFSPS6Q, seg_pos={0 0}, upload=81860]: Done uploading piece. err: <nil> Jul 28 00:55:56.974658303 (85h20m6.697221343s) [noise=true , address=104.36.82.53:4128 , piece_id=WABIAO3HBNS4NYPDI2S2JQILBR66ISPJ5URZ7XWKRXJ2XITH4EKQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 188.166.225.170:33596->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974705122 (85h20m6.697268163s) [noise=true , address=104.36.82.10:4087 , piece_id=EW4ILHZTBNYE3SK627RTBZZVHO2TP3X3JFMGV2LCCCIJ5E2YOKRQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:4087): protocol: rpc: dial tcp 104.36.82.10:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974739258 (85h20m6.697302298s) [noise=true , address=104.36.82.56:4233 , piece_id=OADJE5KYPFDQAZGA6HYMKD2FTGTUOLOHSDMUS7NSOSN5SXHQYLVQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974762572 (85h20m6.697325612s) [noise=true , address=104.36.82.4:4813 , piece_id=EVVOMMGNKU6SEIUGXB2J4RWGPTJGQ6TKL7GN7FRRWKRQJXUCTBDA, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:104.36.82.4:4813): protocol: rpc: dial tcp 104.36.82.4:4813: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974797950 (85h20m6.69736099s) [noise=true , address=104.36.82.13:4199 , piece_id=U7HBGTKEODH56SM5RU3FHECB64RWWJL6MKGW3FHHDLJJDJGGWC6A, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7, address:104.36.82.13:4199): protocol: rpc: dial tcp 104.36.82.13:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974824561 (85h20m6.697387601s) [noise=true , address=104.36.82.11:4138 , piece_id=V7YI2ECXX67FILI74NM7LTK5257L7Q573CJHEW5MHM4WSWK5PEFQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974835031 (85h20m6.697398071s) [noise=true , address=104.36.82.42:4817 , piece_id=SO5GWCRUYBSI4PTNNI2565ONEPW6G3VD4NPJNH5RUTAWVZLDI2EQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: rpc: dial tcp 104.36.82.42:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974873064 (85h20m6.697436104s) [noise=true , address=104.36.82.47:4524 , piece_id=2LGWQ56OHU6IMV3IJQGVKNM4BE66MYTHELVQCW2GRAJXHX3GINOA, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: rpc: dial tcp 104.36.82.47:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974876129 (85h20m6.697439169s) [noise=true , address=104.36.82.49:4011 , piece_id=FMZJXIHC7EXZFE5TZK3I3IT7ZED2REBHH7I7J5CLOFXSDCSI2YHQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1DV1WUL3dGBjH23a6xtetmqmLE9fER1FQf53gmYFxZiP25ra8N, address:104.36.82.49:4011): protocol: rpc: dial tcp 104.36.82.49:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974903682 (85h20m6.697466722s) [noise=true , address=104.36.82.54:4185 , piece_id=LTPYIGXFB3SGAJF2YA7ZALRIPTJ3OLZL22ZWIW6DP7D6OHHDHCYA, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1cLdyyVLrp1yBmgsfBrfYc6N2RFbBichYoBbo2F7EZfaBXtkfk, address:104.36.82.54:4185): protocol: rpc: dial tcp 104.36.82.54:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974911768 (85h20m6.697474808s) [noise=true , address=104.36.82.11:4148 , piece_id=CQ2C5USTWRRL6BZX4U4ESHUHK7USH7KRIJSTEGBJ5XIMOIWJB6DA, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:12Rkh9uZUA5jXBVZvvt1jE7GHiEszKvR9PweKEry2AkdwoCiANs, address:104.36.82.11:4148): protocol: rpc: dial tcp 104.36.82.11:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974921867 (85h20m6.697484907s) [noise=true , address=104.36.82.14:4236 , piece_id=VLT4LD7V3DCPZMHV4ZC6S2KABEGQBZBITFUVMY4XCXJLVJIIYR4A, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1XaE29W27ym2RmuGdBvHeKcQZZEUbask1zsm9Bz8xLokZNGfcj, address:104.36.82.14:4236): protocol: rpc: dial tcp 104.36.82.14:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974935683 (85h20m6.697498723s) [noise=true , address=104.36.82.12:4158 , piece_id=XUWVKRTXCMDW5JM7BBQIY4EV3YY5MMDOTX6NCNUN4UM2ZJXN4BYA, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:4158): protocol: rpc: dial tcp 104.36.82.12:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974969679 (85h20m6.697532719s) [noise=true , address=104.36.82.3:4796 , piece_id=RKRRWWZCWAXZCC3LGDGUXDB2DKXC3NSVX3427PV3VKMVT36DUHUQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: rpc: dial tcp 104.36.82.3:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974975500 (85h20m6.69753854s) [noise=true , address=104.36.82.15:4284 , piece_id=AQ333IVRVHM6MHWLPOPSKKQHX5FQFN555VSYD2SQR6ULQIPPYEVA, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1YAnLu8onh19Mx524VikQzCHEFW4U8dgtuLx8amtQAiReSCisF, address:104.36.82.15:4284): protocol: rpc: dial tcp 104.36.82.15:4284: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.974985809 (85h20m6.697548849s) [noise=true , address=104.36.82.21:4480 , piece_id=ETBMOEKEKAPFZRYW3GMT5GAM27ALSATKQVXKD7SIDET46TRSXFUQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6, address:104.36.82.21:4480): protocol: rpc: dial tcp 104.36.82.21:4480: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:55:56.975065322 (85h20m6.697628362s) [noise=true , address=104.36.82.52:4103 , piece_id=2VEIDUA22WA2SVC4X5FWG7J46OWKUPVFJBKBDQGYNKU5ZQPW3DFQ, seg_pos={0 0}, upload=81860]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:55:56.975080411 (85h20m6.697643441s) [seg_pos={0 0}, upload=81860]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:55:57.205516900 (85h20m6.92807994s) [upload=81860]: Done uploading segments... Jul 28 00:55:57.205541968 (85h20m6.928105008s) [upload=81860]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.233139914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.X..........X.....X.X...........X...................X....] duration: 5.749962554s FAIL (2025-07-28 00:55:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.749985838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.......X.XX....X....X.......X.....X.....................] duration: 5.815451674s FAIL (2025-07-28 00:55:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.815480329s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................X..................................] duration: 1.841108938s FAIL (2025-07-28 00:55:07 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.841172701s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [...........X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23043576s FAIL (2025-07-28 00:54:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== .. Jul 28 00:53:57.628167482 (85h18m7.350730522s) [noise=true , address=104.36.82.21:4508 , piece_id=L6QHYBRMETGENWZBIGB4W5USSDIRLE4TMS3YNXI5RXOATBSJSPXQ, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:57.629174760 (85h18m7.3517378s) [noise=true , address=104.36.82.2:4760 , piece_id=ZWZS32NBI5YANFACH5HK6BTYDLTD6B26HEXCNNMYVRNYW7QNB5PQ, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:57.629484382 (85h18m7.352047422s) [noise=true , address=104.36.82.14:4225 , piece_id=Z2ZUOUZJZ4XKILQCWSUPQBQCTIVZ4Q3DV5S5WVWSWHZFWSBHX3DA, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:57.629720824 (85h18m7.352283865s) [noise=true , address=104.36.82.12:4180 , piece_id=JA6Y5DTLIS34SLCIQHQYKNAHMA5F32XSJBNMNJVF7ZRH6LEQ3SBQ, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:57.630265556 (85h18m7.352828596s) [noise=true , address=104.36.82.15:4285 , piece_id=URB43AMIUIUUOZAM37SL42H5FUOFIMX7B7FLEB3TIA6E52QUCPXQ, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:59.149248104 (85h18m8.871811134s) [upload=81845]: Next returned nil segment Jul 28 00:53:59.149266138 (85h18m8.871829178s) [upload=81845]: Waiting for error group managing segments... Jul 28 00:54:02.641709492 (85h18m12.364272532s) [noise=true , address=79.127.201.215:4347 , piece_id=2LIPMUCFKL2544IMN7AUC4DRFSIEWJCL7OSZMUT5IYJVC7OTP7GQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: <nil> Jul 28 00:54:56.515684334 (85h19m6.238247444s) [noise=true , address=104.36.82.16:4320 , piece_id=32OP2NA6YWLWZX2QFFYSGPSWNO33WP2GV7T3VRP247XH4UW25EIA, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B, address:104.36.82.16:4320): protocol: write tcp 188.166.225.170:32954->104.36.82.16:4320: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.515777432 (85h19m6.238340492s) [noise=true , address=104.36.82.10:4109 , piece_id=C4OFGLUGO7URQZYVQWBVL5FTMEHT7HE3ZKPSV7HQ7LIBOSYFIL5Q, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12ZE1ut9Tum4tzq2m8zbFaGUNLdNZSZQYARkayDAfE1b7uZtWjW, address:104.36.82.10:4109): protocol: write tcp 188.166.225.170:57884->104.36.82.10:4109: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.515862555 (85h19m6.238425645s) [noise=true , address=104.36.82.6:4905 , piece_id=XURLVQTPGGAJI4REOW7UELMFX5NV7AEKMQMTGSU3PDTENMO4ADWQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1qAL8paQ8Z3A9VgPRQ2UUG2LytmYZKdz9hjLzK8w8fQmHiYf88, address:104.36.82.6:4905): protocol: write tcp 188.166.225.170:47648->104.36.82.6:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.515938951 (85h19m6.238502081s) [noise=true , address=104.36.82.11:4117 , piece_id=4BO72GMCSVO32JFUQOPMNUASS6E5IS3QFIXNXHMKGOATDPARHQSQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the, address:104.36.82.11:4117): protocol: write tcp 188.166.225.170:44712->104.36.82.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516023292 (85h19m6.238586373s) [noise=true , address=104.36.82.41:4792 , piece_id=UHLYGW6R74SOLE5MFIHQL242RATJPYMUN2IL6IZBK4PAWB4VVY6Q, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: write tcp 188.166.225.170:36322->104.36.82.41:4792: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516085311 (85h19m6.238648341s) [noise=true , address=104.36.82.2:4760 , piece_id=ZWZS32NBI5YANFACH5HK6BTYDLTD6B26HEXCNNMYVRNYW7QNB5PQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:104.36.82.2:4760): protocol: write tcp 188.166.225.170:54414->104.36.82.2:4760: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516129977 (85h19m6.238693037s) [noise=true , address=104.36.82.15:4285 , piece_id=URB43AMIUIUUOZAM37SL42H5FUOFIMX7B7FLEB3TIA6E52QUCPXQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y, address:104.36.82.15:4285): protocol: write tcp 188.166.225.170:36972->104.36.82.15:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516138362 (85h19m6.238701412s) [noise=true , address=104.36.82.21:4508 , piece_id=L6QHYBRMETGENWZBIGB4W5USSDIRLE4TMS3YNXI5RXOATBSJSPXQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:104.36.82.21:4508): protocol: write tcp 188.166.225.170:56478->104.36.82.21:4508: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516173851 (85h19m6.238736891s) [noise=true , address=104.36.82.52:4112 , piece_id=NQGLXVN5VABGY5WEV62VGTCEQAHIA6KR6VWL657C33NU7M3XFYXA, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: write tcp 188.166.225.170:36752->104.36.82.52:4112: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516210361 (85h19m6.238773421s) [noise=true , address=104.36.82.22:4516 , piece_id=YPVSAKV3HE5JRBL664F2XY2CUFKAJNUXX5YJOECGAFJRHW64A24A, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:104.36.82.22:4516): protocol: write tcp 188.166.225.170:34646->104.36.82.22:4516: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516221221 (85h19m6.238784271s) [noise=true , address=104.36.82.55:4194 , piece_id=IK2HRKYBVPHBMXVXQOYG5O2BD5BBMTUPNVRSNVM3Z5QBOKBQDEQQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: write tcp 188.166.225.170:42578->104.36.82.55:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516263011 (85h19m6.238826071s) [noise=true , address=104.36.82.14:4225 , piece_id=Z2ZUOUZJZ4XKILQCWSUPQBQCTIVZ4Q3DV5S5WVWSWHZFWSBHX3DA, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:104.36.82.14:4225): protocol: write tcp 188.166.225.170:51994->104.36.82.14:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516306754 (85h19m6.238869795s) [noise=true , address=104.36.82.12:4180 , piece_id=JA6Y5DTLIS34SLCIQHQYKNAHMA5F32XSJBNMNJVF7ZRH6LEQ3SBQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: write tcp 188.166.225.170:44012->104.36.82.12:4180: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516330119 (85h19m6.238893159s) [noise=true , address=104.36.82.13:4219 , piece_id=S357VGA6B3ZQ2TD25THRYAWYV4LORN3IG2AW3M54SCJRIN2UDZ7A, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12fUzLg93yKvyKkNvuvcKZHyvJ2HGE6tUCg1Wm7ZZjKKeDZDWm9, address:104.36.82.13:4219): protocol: write tcp 188.166.225.170:44550->104.36.82.13:4219: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516372701 (85h19m6.238935731s) [noise=true , address=104.36.82.22:4533 , piece_id=QOTOPEX2CQXCARA2B7X3UBN2PEZBXG2A22AXN3766VONJMA2HOSQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12LXK5pbXavbPZh4YSuEkgk7wsRqwvcVEHv4w9FDRYqSkEU89VF, address:104.36.82.22:4533): protocol: write tcp 188.166.225.170:50010->104.36.82.22:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516383341 (85h19m6.238946381s) [noise=true , address=104.36.82.51:4078 , piece_id=4AEEVMRRMBVSEDKW6FJQ4LGDOX36MFP2UGFNHD4VGXOKPTWIUATQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd, address:104.36.82.51:4078): protocol: write tcp 188.166.225.170:46132->104.36.82.51:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516389613 (85h19m6.238952653s) [noise=true , address=104.36.82.47:4528 , piece_id=MUWLZWYALT6OUZK4P3TGAWDKI6SFY5V3MQJHVES5XNTLRIG5HN5Q, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: write tcp 188.166.225.170:43538->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516446251 (85h19m6.239009281s) [seg_pos={0 0}, upload=81845]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:54:56.744110367 (85h19m6.466673397s) [upload=81845]: Done uploading segments... Jul 28 00:54:56.744135285 (85h19m6.466698325s) [upload=81845]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.230508429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X..........X.....X.X...........X...................X...] duration: 5.822676939s FAIL (2025-07-28 00:54:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.822703959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.......X.XX....X....X.......X.....X....................] duration: 5.725334524s FAIL (2025-07-28 00:54:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.725360955s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................X.................................] duration: 1.891316039s FAIL (2025-07-28 00:53:57 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.891339043s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [............X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235751492s FAIL (2025-07-28 00:53:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ading piece... Jul 28 00:52:57.156942030 (85h17m6.87950507s) [noise=true , address=104.36.82.5:4864 , piece_id=2FR6IDHUOG67JLVOCGBSHMFFPZJI77ECI45ZY6FRKGEV32BZSRTQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157071438 (85h17m6.879634468s) [noise=true , address=104.36.82.15:4281 , piece_id=VM3KYKNERRFAUI5ZXDLMRLSVFG37UPCV5KYKEM63JP6UURP5X44Q, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157151572 (85h17m6.879714612s) [noise=true , address=79.127.201.215:4347 , piece_id=6GUNVFTJFOB4NCFUQJ3AN7WQBODY236O3LGQFUSXY7PG36UPKZTQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157405357 (85h17m6.879968387s) [noise=true , address=104.36.82.39:4735 , piece_id=AM74U7EJ5CF3YYHPM7TIB4XLGLV7VLQOSAX72XR4CRWP5CO7OKDA, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157416599 (85h17m6.879979639s) [noise=true , address=104.36.82.13:4203 , piece_id=EU7Z7XATZTTX2DCE7KFSK5WBAIWJCBB5SYLZ4HKWZ5FF2CM5RRFQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157833877 (85h17m6.880396907s) [noise=true , address=104.36.82.21:4503 , piece_id=WQUEBRVA75MWPVFVIVYM5ZNEKS6GPOJYLPCWKMDMQ2JNOVCKJ4BQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.158381635 (85h17m6.880944675s) [noise=true , address=104.36.82.20:4454 , piece_id=LIKII3N3JAF2F6SRGVPQ6BPHDG64B5QZPOLBCFO3CMF7T3DHW6AQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.158865310 (85h17m6.88142835s) [noise=true , address=104.36.82.11:4145 , piece_id=YJ3F7JXTTVRYCZPXLM7YLSHFRTNHOM7TSYXTTU4TY4OLFE6LPMWQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.159322155 (85h17m6.881885195s) [noise=true , address=104.36.82.47:4517 , piece_id=M74A7A26ARCRGASU626SHFJEYAUZ2PHOPRMVLTSXWFG7ZRFT4UPA, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:58.618677399 (85h17m8.341240439s) [upload=81829]: Next returned nil segment Jul 28 00:52:58.618699211 (85h17m8.341262251s) [upload=81829]: Waiting for error group managing segments... Jul 28 00:53:01.801272985 (85h17m11.523836045s) [noise=true , address=79.127.201.215:4347 , piece_id=6GUNVFTJFOB4NCFUQJ3AN7WQBODY236O3LGQFUSXY7PG36UPKZTQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: <nil> Jul 28 00:53:03.216039458 (85h17m12.938602498s) [noise=true , address=64.49.13.140:4509 , piece_id=GTWJTOVM6PRC6F3I26DPWCG5SUNO3C2QSF2NHGO4YCVAHCKBEICQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: <nil> Jul 28 00:53:56.050333153 (85h18m5.772896193s) [noise=true , address=104.36.82.45:4920 , piece_id=PHTGUIMR3UXAVH7OA6LUARLUZRP35BB53SDS3DKFY5LCNSRFWQZA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S, address:104.36.82.45:4920): protocol: rpc: dial tcp 104.36.82.45:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050374954 (85h18m5.772937994s) [noise=true , address=104.36.82.56:4242 , piece_id=7WHETCKOD7LFXFQMECEJYULOFQRWAXNCTUJBD3LOV45Z65LVN7FA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: rpc: dial tcp 104.36.82.56:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050404460 (85h18m5.7729675s) [noise=true , address=104.36.82.12:4159 , piece_id=SJKCI2LWPEC5CMPF23HWM4BUTTRKKPJRC2UITNVJUUL3HB5VSIIQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: rpc: dial tcp 104.36.82.12:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050433325 (85h18m5.772996365s) [noise=true , address=104.36.82.13:4203 , piece_id=EU7Z7XATZTTX2DCE7KFSK5WBAIWJCBB5SYLZ4HKWZ5FF2CM5RRFQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:18cNSsgqn3n6z5LxRcVCRzDzfFDpvptV1MKh6NdNRLW4bg9fWM, address:104.36.82.13:4203): protocol: rpc: dial tcp 104.36.82.13:4203: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050459796 (85h18m5.773022836s) [noise=true , address=104.36.82.21:4503 , piece_id=WQUEBRVA75MWPVFVIVYM5ZNEKS6GPOJYLPCWKMDMQ2JNOVCKJ4BQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:17F3Gn5LYgekp8WgagEdFXtRu8jCc27qA9UuW7pWsJ9FUFKFSi, address:104.36.82.21:4503): protocol: rpc: dial tcp 104.36.82.21:4503: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050509121 (85h18m5.773072161s) [noise=true , address=104.36.82.53:4154 , piece_id=NDUJHIYDPNNFGQRCYOY4PJHPBTME6YZZSFYSD5J3FEMMAZZO224A, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050550409 (85h18m5.773113449s) [noise=true , address=104.36.82.39:4735 , piece_id=AM74U7EJ5CF3YYHPM7TIB4XLGLV7VLQOSAX72XR4CRWP5CO7OKDA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1QCQZ8eFGLa5TAaAJJzUip8o5DAELzTBTm4UFbt3Am5yYWXAZF, address:104.36.82.39:4735): protocol: rpc: dial tcp 104.36.82.39:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050578974 (85h18m5.773142004s) [noise=true , address=104.36.82.20:4454 , piece_id=LIKII3N3JAF2F6SRGVPQ6BPHDG64B5QZPOLBCFO3CMF7T3DHW6AQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:104.36.82.20:4454): protocol: rpc: dial tcp 104.36.82.20:4454: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050609382 (85h18m5.773172422s) [noise=true , address=104.36.82.47:4517 , piece_id=M74A7A26ARCRGASU626SHFJEYAUZ2PHOPRMVLTSXWFG7ZRFT4UPA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050639088 (85h18m5.773202129s) [noise=true , address=104.36.82.55:4228 , piece_id=KQPAJIBJRSUOTHLU7B4LXGP7HJWLQ5DWA3XIYOCNAMZ4ECXDV2OQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: rpc: dial tcp 104.36.82.55:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050665539 (85h18m5.773228589s) [noise=true , address=104.36.82.11:4145 , piece_id=YJ3F7JXTTVRYCZPXLM7YLSHFRTNHOM7TSYXTTU4TY4OLFE6LPMWQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: rpc: dial tcp 104.36.82.11:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050679336 (85h18m5.773242376s) [noise=true , address=104.36.82.15:4281 , piece_id=VM3KYKNERRFAUI5ZXDLMRLSVFG37UPCV5KYKEM63JP6UURP5X44Q, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2, address:104.36.82.15:4281): protocol: rpc: dial tcp 104.36.82.15:4281: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050695446 (85h18m5.773258486s) [noise=true , address=104.36.82.14:4241 , piece_id=F3LNXFISZEFQMYJPMOPHB5WSTO2H5NUGUAT2KKZKYZSBG6RDG3EA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12aBy6nQScbjzmBSScfEZwTX8NCcP9c71FRQkUH54WvZhhhtXLR, address:104.36.82.14:4241): protocol: rpc: dial tcp 104.36.82.14:4241: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050711217 (85h18m5.773274257s) [noise=true , address=104.36.82.7:4943 , piece_id=EA7LC6TR7VJME2LQIBJGQNYDGOTHDYBD2GECX5KGPWSVW2PSYRDA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12BdCdPHn6ko7nfY4sPuc4NutXABzJDdHHhtSnyXri5gofhZEZJ, address:104.36.82.7:4943): protocol: write tcp 188.166.225.170:39156->104.36.82.7:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050722879 (85h18m5.773285919s) [noise=true , address=104.36.82.49:4999 , piece_id=PGUGUPCSP2PWCHXWZNZGRX7T3XFOWOF7DTKPBTMHFKE3WVTLO7AQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1tSiyPFy1xn1YRCLmKmPmbTm3LUmDP9nT1YRCXnsuyY6XsLKrG, address:104.36.82.49:4999): protocol: rpc: dial tcp 104.36.82.49:4999: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050741775 (85h18m5.773304805s) [noise=true , address=104.36.82.5:4864 , piece_id=2FR6IDHUOG67JLVOCGBSHMFFPZJI77ECI45ZY6FRKGEV32BZSRTQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: write tcp 188.166.225.170:39984->104.36.82.5:4864: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050759730 (85h18m5.77332277s) [seg_pos={0 0}, upload=81829]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:53:56.284294548 (85h18m6.006857588s) [upload=81829]: Done uploading segments... Jul 28 00:53:56.284320297 (85h18m6.006883327s) [upload=81829]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.235767333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X..........X.....X.X...........X...................X..] duration: 5.871791097s FAIL (2025-07-28 00:53:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.871818689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X.......X.XX....X....X.......X.....X...................] duration: 5.76684137s FAIL (2025-07-28 00:53:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.766868079s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................X................................] duration: 1.839691876s FAIL (2025-07-28 00:53:28 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.839717235s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.......X.XX....X....X.......X.....X..................] duration: 5.806881307s FAIL (2025-07-28 00:53:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.806910833s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.X..........X.....X.X...........X...................X.] duration: 5.770423552s FAIL (2025-07-28 00:52:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.770441186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [.............X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238488656s FAIL (2025-07-28 00:52:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 0:51:56.628136725 (85h16m6.350699765s) [noise=true , address=104.36.82.15:4269 , piece_id=6AUB6RHUX75GUGKGWCG4DOJGOALIJIUPRP6IGHA5QUV2TGBUNFFA, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.628529166 (85h16m6.351092206s) [noise=true , address=104.36.82.52:4088 , piece_id=NTM2SUYNUUCGGRWJGEJBZZSYFNV77ZPNL45QVMEL7VF3EZG6OULQ, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.628785516 (85h16m6.351348546s) [noise=true , address=104.36.82.12:4161 , piece_id=W2GXL43MFJRVHISR3BG3GUE37ESM324WZ4BETSXEFHIQUMS7NM4A, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.628990168 (85h16m6.351553198s) [noise=true , address=104.36.82.13:4205 , piece_id=XWOXWEZYH7DKMK3L7HN6BDICKZUID3NC2BCX2CHSI6JCLVV6O32A, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.629227502 (85h16m6.351790542s) [noise=true , address=104.36.82.53:4147 , piece_id=AMLUVXYQ6TQGLC2MZZSKCNQ5GSR6TL35ASZTZ64RQF3X5NMJM3GQ, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.629487329 (85h16m6.352050369s) [noise=true , address=104.36.82.56:4264 , piece_id=MJZCE23A5F6ILUNLNYSWHB3TXYKCAAQGZ7J372NT3V7RMCRBPYEQ, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.629683274 (85h16m6.352246304s) [noise=true , address=104.36.82.44:4894 , piece_id=QM3BMURZGNFIFLEGFZE5UH5TBQE6HQGI32YSU7VNATR7DHMZZI3Q, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.629939534 (85h16m6.352502574s) [noise=true , address=64.49.13.137:4498 , piece_id=AONBCU6MLEV64NEFBACPQTKA57IPA4GE35K32U4O56X655PDVKTA, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:57.751189877 (85h16m7.473752917s) [upload=81816]: Next returned nil segment Jul 28 00:51:57.751201469 (85h16m7.473764509s) [upload=81816]: Waiting for error group managing segments... Jul 28 00:51:58.033568719 (85h16m7.756131759s) [noise=true , address=64.49.13.137:4498 , piece_id=AONBCU6MLEV64NEFBACPQTKA57IPA4GE35K32U4O56X655PDVKTA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: <nil> Jul 28 00:52:55.498735625 (85h17m5.221298665s) [noise=true , address=104.36.82.21:4501 , piece_id=QSOSARYHH577ULBMYT3RHSQYCC4VKTLZ5GFOFG5VHQQGOSHDUFIA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1e1R3gjh5CYJdDqck7xcR6A19Hb443FWbX8X2AQikxt5f4AVvU, address:104.36.82.21:4501): protocol: rpc: dial tcp 104.36.82.21:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498781994 (85h17m5.221345034s) [noise=true , address=104.36.82.55:4223 , piece_id=UIGTXCPQHX6N37PDDYLYSR3BJJIU3VQEXBX5E4NIVFOHNQB6TWVQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: rpc: dial tcp 104.36.82.55:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498818143 (85h17m5.221381173s) [noise=true , address=104.36.82.55:4211 , piece_id=BQ7B5C4W5743MQUY5SMTUAFLDKGWIQI4JPFVHQTZICPZTAQIMM3Q, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: rpc: dial tcp 104.36.82.55:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498831679 (85h17m5.221394719s) [noise=true , address=104.36.82.47:4534 , piece_id=XX4Y5GWLLITF2YKN2QBMK2FRGJ47ONITXTJEDFA7QDJF4JCXXLQA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: rpc: dial tcp 104.36.82.47:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498849933 (85h17m5.221412973s) [noise=true , address=104.36.82.10:4088 , piece_id=NWJRJAN5HYHAZ234NMP2OEFQ34UUJA3MZVQEKSKHKNACHBQLGN3Q, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, address:104.36.82.10:4088): protocol: rpc: dial tcp 104.36.82.10:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498886244 (85h17m5.221449274s) [noise=true , address=104.36.82.4:4824 , piece_id=E5YUSGVBFZ6A7X2D3AM26PA6HI27KDFB56444PETMNMUM7PDJQDA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498893026 (85h17m5.221456066s) [noise=true , address=104.36.82.14:4244 , piece_id=2A7OABT2E3EUQXFOWXBRHP3X5BCHV2JPVJOTZRJFIMA7PTOSIFVA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:4244): protocol: rpc: dial tcp 104.36.82.14:4244: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498924586 (85h17m5.221487627s) [noise=true , address=104.36.82.2:4762 , piece_id=BC7J47MJNZELCKRHDXJL3F6YVOGIHZCCNF7O7R4ZCMMCMA6J5CAA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498935878 (85h17m5.221498918s) [noise=true , address=104.36.82.11:4143 , piece_id=DBP7SN5QO7USRS4X6OCGETT72UZGWQZ66K4LFFAKFPQOZLCR5MUA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: rpc: dial tcp 104.36.82.11:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498960625 (85h17m5.221523665s) [noise=true , address=104.36.82.56:4264 , piece_id=MJZCE23A5F6ILUNLNYSWHB3TXYKCAAQGZ7J372NT3V7RMCRBPYEQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498979241 (85h17m5.221542281s) [noise=true , address=104.36.82.52:4088 , piece_id=NTM2SUYNUUCGGRWJGEJBZZSYFNV77ZPNL45QVMEL7VF3EZG6OULQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:104.36.82.52:4088): protocol: rpc: dial tcp 104.36.82.52:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498992376 (85h17m5.221555416s) [noise=true , address=104.36.82.44:4894 , piece_id=QM3BMURZGNFIFLEGFZE5UH5TBQE6HQGI32YSU7VNATR7DHMZZI3Q, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499017654 (85h17m5.221580694s) [noise=true , address=104.36.82.12:4161 , piece_id=W2GXL43MFJRVHISR3BG3GUE37ESM324WZ4BETSXEFHIQUMS7NM4A, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1SdSwg6tZCKSturNytQmWUmSWQ2G5dUSrb2RA8hxLLpBtBE6tf, address:104.36.82.12:4161): protocol: rpc: dial tcp 104.36.82.12:4161: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499024338 (85h17m5.221587378s) [noise=true , address=104.36.82.54:4165 , piece_id=HYLWQ4OJ2RI5H7V7IHMLOAF6L22BMOPHVJ77N2EBSCEO3PAI4JYQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499059004 (85h17m5.221622044s) [noise=true , address=104.36.82.15:4269 , piece_id=6AUB6RHUX75GUGKGWCG4DOJGOALIJIUPRP6IGHA5QUV2TGBUNFFA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12V8v5nzyyouSqndATi4fRJUC6HgFbhguBjnmwpki48zdRBTcgC, address:104.36.82.15:4269): protocol: rpc: dial tcp 104.36.82.15:4269: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499069994 (85h17m5.221633034s) [noise=true , address=104.36.82.13:4205 , piece_id=XWOXWEZYH7DKMK3L7HN6BDICKZUID3NC2BCX2CHSI6JCLVV6O32A, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr, address:104.36.82.13:4205): protocol: write tcp 188.166.225.170:52366->104.36.82.13:4205: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499107737 (85h17m5.221670777s) [noise=true , address=104.36.82.53:4147 , piece_id=AMLUVXYQ6TQGLC2MZZSKCNQ5GSR6TL35ASZTZ64RQF3X5NMJM3GQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 188.166.225.170:52012->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499135220 (85h17m5.22169826s) [seg_pos={0 0}, upload=81816]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:52:55.734778493 (85h17m5.457341533s) [upload=81816]: Done uploading segments... Jul 28 00:52:55.734803110 (85h17m5.45736614s) [upload=81816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.238514675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.X..........X.....X.X...........X...................X] duration: 30.232625465s FAIL (2025-07-28 00:52:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232646775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X.......................X...............................] duration: 1.820323489s FAIL (2025-07-28 00:52:46 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.820348719s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.......X.XX....X....X.......X.....X.................] duration: 5.825094181s FAIL (2025-07-28 00:52:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.82511987s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [..............X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230475841s FAIL (2025-07-28 00:51:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== , piece_id=FM6XCLSEDVSO6D7IQ66PLBS4Q2KCX6JYHYD3L377OYWROWSKKGHQ, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.671947809 (85h15m2.394510849s) [noise=true , address=104.36.82.6:4899 , piece_id=X4ZRUYPCSO3HO4AITFBYCAKBMRFDSGWFQVO4ONB2RRZGT2MUAR7A, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.672152380 (85h15m2.39471542s) [noise=true , address=104.36.82.21:4507 , piece_id=4KKEMFBCU5FVZ4LFXAU4E6PTE3DSPNUXG3GZYEPVGWJ6MGPBNMGA, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.672513801 (85h15m2.395076841s) [noise=true , address=104.36.82.2:4740 , piece_id=5SSP27FUANOLHIYTFJDDWLAK2KJ7GKW5A36WYRWTFERJIQDLVFUQ, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.672769250 (85h15m2.39533229s) [noise=true , address=104.36.82.12:4176 , piece_id=XSY5C7OJJXYLU7W2OTEGPOEVTJGIH2UVJQQ2SAL3ZGLUKSNONF7A, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.672962099 (85h15m2.395525139s) [noise=true , address=104.36.82.49:4001 , piece_id=TGTXQCFPUR4CKR4BMUM47M3WBCMWVPOTVC4EMBQCIAF6MWW4HTGQ, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.673148186 (85h15m2.395711226s) [noise=true , address=104.36.82.55:4199 , piece_id=6I7E4GUWCHEIDPKHP34CJPIW5KJ3WMPZMZHDHL7TC6QASBJJH5YQ, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:53.779601427 (85h15m3.502164467s) [upload=81796]: Next returned nil segment Jul 28 00:50:53.779618670 (85h15m3.50218173s) [upload=81796]: Waiting for error group managing segments... Jul 28 00:50:54.091618936 (85h15m3.814182016s) [noise=true , address=79.127.205.235:4673 , piece_id=HGYTP5HVHLUOLIM6HTBNAHVT4XAJJG4A3SM7ZKJ7DRZK3GTVJPFA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: <nil> Jul 28 00:50:56.527577110 (85h15m6.25014015s) [noise=true , address=79.127.201.215:4356 , piece_id=ANJOP5GGNX3YB3WBFUHXLMPXKLNXZRLRVKC6FKHAYMJW5CC253WA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: <nil> Jul 28 00:51:51.558091026 (85h16m1.280654066s) [noise=true , address=104.36.82.15:4280 , piece_id=FM6XCLSEDVSO6D7IQ66PLBS4Q2KCX6JYHYD3L377OYWROWSKKGHQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1t4hMpMGCu1YMmNLX6fssTjWy9Xi5Fp8kdHTDmkhxH9qsZiYM6, address:104.36.82.15:4280): protocol: write tcp 188.166.225.170:41194->104.36.82.15:4280: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558135882 (85h16m1.280698922s) [noise=true , address=104.36.82.21:4507 , piece_id=4KKEMFBCU5FVZ4LFXAU4E6PTE3DSPNUXG3GZYEPVGWJ6MGPBNMGA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: write tcp 188.166.225.170:55388->104.36.82.21:4507: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558178855 (85h16m1.280741895s) [noise=true , address=104.36.82.12:4176 , piece_id=XSY5C7OJJXYLU7W2OTEGPOEVTJGIH2UVJQQ2SAL3ZGLUKSNONF7A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m, address:104.36.82.12:4176): protocol: rpc: dial tcp 104.36.82.12:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558207449 (85h16m1.280770489s) [noise=true , address=104.36.82.53:4152 , piece_id=IE7OPN62IUI2RKWI7ZVZFMRTG3KP2FARILU7ZV3LJAG65QDLS36Q, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 188.166.225.170:43414->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558237066 (85h16m1.280800096s) [noise=true , address=104.36.82.5:4860 , piece_id=VI6HIYB47YZ2JXCG2UIE3RFX3K2NQFWG74CSVCIWOPMH67JKRM4A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: write tcp 188.166.225.170:37850->104.36.82.5:4860: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558294115 (85h16m1.280857155s) [noise=true , address=104.36.82.49:4001 , piece_id=TGTXQCFPUR4CKR4BMUM47M3WBCMWVPOTVC4EMBQCIAF6MWW4HTGQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1iZnfkAxoot7KjtjYbxbSFvDhDFcLd8M63Qkh67GuMXMJjdruD, address:104.36.82.49:4001): protocol: write tcp 188.166.225.170:46144->104.36.82.49:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558349371 (85h16m1.280912421s) [noise=true , address=104.36.82.52:4086 , piece_id=CSC745NAKWCFE3GI66GZM3FAT734HQMMACRSHXTMP3BCTQZYV2HA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:104.36.82.52:4086): protocol: write tcp 188.166.225.170:44190->104.36.82.52:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558408944 (85h16m1.280971994s) [noise=true , address=104.36.82.11:4142 , piece_id=7J35VV7ENERICAC4GBD7G2GVTSFX56GTIOPGMGX5V3HIHI26IBKQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: write tcp 188.166.225.170:39942->104.36.82.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558454442 (85h16m1.281017482s) [noise=true , address=104.36.82.54:4190 , piece_id=MPRMQJYNI24CLMP5J3DNEYNQZWHUF67ERGX5L5YB5XT4Q2IWIWHQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: write tcp 188.166.225.170:39452->104.36.82.54:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558461024 (85h16m1.281024064s) [noise=true , address=104.36.82.6:4899 , piece_id=X4ZRUYPCSO3HO4AITFBYCAKBMRFDSGWFQVO4ONB2RRZGT2MUAR7A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 188.166.225.170:48118->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558485832 (85h16m1.281048872s) [noise=true , address=104.36.82.2:4740 , piece_id=5SSP27FUANOLHIYTFJDDWLAK2KJ7GKW5A36WYRWTFERJIQDLVFUQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:12wgQDbE17gP8c6kbKnNyywSkY8Hk7mchcMaymLBQzPwRsSB8QE, address:104.36.82.2:4740): protocol: write tcp 188.166.225.170:39742->104.36.82.2:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558512863 (85h16m1.281075903s) [noise=true , address=104.36.82.54:4158 , piece_id=3SW3QX7W3UPFHAS5WEW55TDKDQHFA3E2WS37NY3ATNA4CGDALXOA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: write tcp 188.166.225.170:49416->104.36.82.54:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558516911 (85h16m1.281079941s) [noise=true , address=104.36.82.55:4199 , piece_id=6I7E4GUWCHEIDPKHP34CJPIW5KJ3WMPZMZHDHL7TC6QASBJJH5YQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: write tcp 188.166.225.170:35812->104.36.82.55:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558543111 (85h16m1.281106151s) [noise=true , address=104.36.82.3:4793 , piece_id=TCF77K4ZK3AILD5PNLUXJHI6AAHRM2S4BR6FG3NV7W37ZLJZTR2A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:104.36.82.3:4793): protocol: write tcp 188.166.225.170:41906->104.36.82.3:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558550255 (85h16m1.281113295s) [noise=true , address=104.36.82.56:4231 , piece_id=PXD4O2OMKKUVPDHLVZYYZPFIC5QA44IAFFA7KVLLNBKQLYTIW34A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: write tcp 188.166.225.170:51018->104.36.82.56:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558562058 (85h16m1.281125098s) [noise=true , address=104.36.82.22:4528 , piece_id=7V3ISUWDHTHQXRQ4G3ZQYNURM36YVYAH6Q3ZBJ2QUVNPIW5XJKEQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1nDzCSeAbqSKpZUVGcu4NEj32jsoqojqSpzfpKYoXciXEjsTwo, address:104.36.82.22:4528): protocol: write tcp 188.166.225.170:53542->104.36.82.22:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558590011 (85h16m1.281153051s) [seg_pos={0 0}, upload=81796]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:51:51.786989094 (85h16m1.509552134s) [upload=81796]: Done uploading segments... Jul 28 00:51:51.787013430 (85h16m1.50957647s) [upload=81796]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.230487643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......................X..............................] duration: 1.839018387s FAIL (2025-07-28 00:51:50 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.839047593s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.X..........X.....X.X...........X...................] duration: 5.849208558s FAIL (2025-07-28 00:51:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.849234919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X.......X.XX....X....X.......X.....X................] duration: 5.71676098s FAIL (2025-07-28 00:51:03 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.716789935s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X.......................X.............................] duration: 1.927016072s FAIL (2025-07-28 00:50:56 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.927091887s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [...............X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.243951942s FAIL (2025-07-28 00:50:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 7877s) [noise=true , address=104.36.82.10:4111 , piece_id=XZXIDRKLRL4IHNHGFW2ZCZKIHXKUWW4EF33XSB7KVYP7AN6J3YBA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190002167 (85h14m1.912565207s) [noise=true , address=104.36.82.9:4019 , piece_id=B7T35DNTWGAM4QRETBQFCGQZ2AWTDFDRBJBIG5YFNGGB5ROG7QBQ, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190237006 (85h14m1.912800046s) [noise=true , address=79.127.201.215:4355 , piece_id=HTFDKRMIH6HT25D2VNKNSLFRNGMU23FZTXUNXBD5LMOU4G63I55Q, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190448781 (85h14m1.913011821s) [noise=true , address=104.36.82.56:4237 , piece_id=BYISKHFOSF5TURCNAUDETJAOBDD2OMBFKFJBSTI52CSGA7YOBMFA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190674222 (85h14m1.913237262s) [noise=true , address=104.36.82.52:4101 , piece_id=MBZ6GPSY4CYARF3WSURA64DTMQT5G6VO6CKZJZTOEHGSEFU6JWOA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190924982 (85h14m1.913488012s) [noise=true , address=104.36.82.13:4216 , piece_id=KCL3AZJWRSP5HNPWVPO5EWPUPTU7UFWNUFIIYFIVNJ3KX6XYOPZA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.191166905 (85h14m1.913729945s) [noise=true , address=104.36.82.54:4180 , piece_id=LONM6HAJLGZMKFPXWVHYM56VYCEGGGHNYYA3SR2KIGD2UACQ2KAQ, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.191382778 (85h14m1.913945818s) [noise=true , address=104.36.82.50:4024 , piece_id=QFIOLHBIXB55PNLXMXCPH5TRWI4NYT7PNOFWS4E5XZ7LNY5SU7OA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:53.686581090 (85h14m3.40914413s) [upload=81779]: Next returned nil segment Jul 28 00:49:53.686598343 (85h14m3.409161383s) [upload=81779]: Waiting for error group managing segments... Jul 28 00:50:00.379103601 (85h14m10.101666641s) [noise=true , address=79.127.201.215:4355 , piece_id=HTFDKRMIH6HT25D2VNKNSLFRNGMU23FZTXUNXBD5LMOU4G63I55Q, seg_pos={0 0}, upload=81779]: Done uploading piece. err: <nil> Jul 28 00:50:51.087461893 (85h15m0.810024933s) [noise=true , address=104.36.82.10:4111 , piece_id=XZXIDRKLRL4IHNHGFW2ZCZKIHXKUWW4EF33XSB7KVYP7AN6J3YBA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:128oYxFebZ8HAoUNDf2Tmqi23Q1kuo2XgzWxKVRCGwaiuZBWtAw, address:104.36.82.10:4111): protocol: rpc: dial tcp 104.36.82.10:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087499605 (85h15m0.810062645s) [noise=true , address=104.36.82.46:4486 , piece_id=JW4K6HTRSUKZZOPH3PR5ATWJYXQT745VYM42D2JW4WP74OVGPGHA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic, address:104.36.82.46:4486): protocol: rpc: dial tcp 104.36.82.46:4486: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087517470 (85h15m0.81008051s) [noise=true , address=104.36.82.56:4237 , piece_id=BYISKHFOSF5TURCNAUDETJAOBDD2OMBFKFJBSTI52CSGA7YOBMFA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: rpc: dial tcp 104.36.82.56:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087535483 (85h15m0.810098524s) [noise=true , address=104.36.82.13:4216 , piece_id=KCL3AZJWRSP5HNPWVPO5EWPUPTU7UFWNUFIIYFIVNJ3KX6XYOPZA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu, address:104.36.82.13:4216): protocol: rpc: dial tcp 104.36.82.13:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087556945 (85h15m0.810119985s) [noise=true , address=104.36.82.2:4756 , piece_id=5DFDMVPOA6L6ZYNXW7LX7PZCMVOI4MMFDPTIA6KXAIBXJ6H6J56A, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1Pow7VVb1PRDBvHjEcS4bfnh3276S313jiGawszkhaJ582pDqQ, address:104.36.82.2:4756): protocol: write tcp 188.166.225.170:35166->104.36.82.2:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087569339 (85h15m0.810132369s) [noise=true , address=104.36.82.53:4154 , piece_id=JUUQPR3EVF7F2U4PJCAN5CFEYE5GQS4WRVCR46INWG2XIR5WTU6A, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087602733 (85h15m0.810165773s) [noise=true , address=104.36.82.54:4180 , piece_id=LONM6HAJLGZMKFPXWVHYM56VYCEGGGHNYYA3SR2KIGD2UACQ2KAQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087633902 (85h15m0.810196942s) [noise=true , address=104.36.82.15:4284 , piece_id=MIGIVE2BV4ZMSAHM77HXQY2BKNROJO6JPY7EFVFPBT4FTRSTAF2A, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1YAnLu8onh19Mx524VikQzCHEFW4U8dgtuLx8amtQAiReSCisF, address:104.36.82.15:4284): protocol: rpc: dial tcp 104.36.82.15:4284: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087659461 (85h15m0.810222511s) [noise=true , address=104.36.82.11:4124 , piece_id=AXSTMSP4XLXO23DC6VXBEH3RD67EABD64ZY353MHIZRAAW3MACYA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1QAgfinEkxYwjzvtr53FRritkoDNbfYAw21qcJuj3uCY4JT4cZ, address:104.36.82.11:4124): protocol: rpc: dial tcp 104.36.82.11:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087684840 (85h15m0.81024788s) [noise=true , address=104.36.82.50:4024 , piece_id=QFIOLHBIXB55PNLXMXCPH5TRWI4NYT7PNOFWS4E5XZ7LNY5SU7OA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1EAeBfX3Xyp8pBDUoS7VhyXwpNbXooSYgxZPxP9kEXT1rZwkoG, address:104.36.82.50:4024): protocol: rpc: dial tcp 104.36.82.50:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087692154 (85h15m0.810255194s) [noise=true , address=104.36.82.9:4019 , piece_id=B7T35DNTWGAM4QRETBQFCGQZ2AWTDFDRBJBIG5YFNGGB5ROG7QBQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:12JuQKLQnXiEcyCj6xthEXSPy2Wu5T1B9m4q9YteoK69rMqVydD, address:104.36.82.9:4019): protocol: rpc: dial tcp 104.36.82.9:4019: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087714757 (85h15m0.810277797s) [noise=true , address=104.36.82.12:4153 , piece_id=FN4ERZAEW44CMJJ4XPDT3IW35Z4VNK5LJPUVHVLRX6CNITCJ5NRQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:124myE1wFhvjA5paJy5tr2hRe8jZCoEdbJLdBwvss1ZnT4GvtZv, address:104.36.82.12:4153): protocol: rpc: dial tcp 104.36.82.12:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087721740 (85h15m0.81028478s) [noise=true , address=104.36.82.52:4101 , piece_id=MBZ6GPSY4CYARF3WSURA64DTMQT5G6VO6CKZJZTOEHGSEFU6JWOA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 188.166.225.170:57606->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087727952 (85h15m0.810290982s) [noise=true , address=104.36.82.41:4768 , piece_id=OONZCR7MTTKBAXGBBCTEFWCOWHQRSDT2DNDMMR2TIHOF5AKYDRWA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087735858 (85h15m0.810298898s) [noise=true , address=104.36.82.3:4792 , piece_id=4RN7MWU5AFIDHHLE4773C4AGETWZWLMNEDTR3VCJG4SBX6MD7Z4Q, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1fokaL7frEkFdVZRGmqaq3TmZVNYpt4agWCJZ8ABHfsYwkjAuv, address:104.36.82.3:4792): protocol: rpc: dial tcp 104.36.82.3:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087741538 (85h15m0.810304578s) [noise=true , address=104.36.82.47:4537 , piece_id=UYZOMZFYI62UIZJFIZNBA6KYPW65MBSIGZTQWPTSB5AYG5NTJPLQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087744293 (85h15m0.810307333s) [noise=true , address=104.36.82.42:4829 , piece_id=EOF4BV45UFKWRBQTQWRLBLUPP5G67NWAXNENKI7ZCPCNB2R66ACQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: write tcp 188.166.225.170:43972->104.36.82.42:4829: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087768029 (85h15m0.810331069s) [seg_pos={0 0}, upload=81779]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:50:51.329260964 (85h15m1.051824004s) [upload=81779]: Done uploading segments... Jul 28 00:50:51.329282675 (85h15m1.051845715s) [upload=81779]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.24396658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.X..........X.....X.X...........X..................] duration: 5.839770412s FAIL (2025-07-28 00:50:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.839808255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X.......X.XX....X....X.......X.....X...............] duration: 5.820750402s FAIL (2025-07-28 00:50:47 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.820779757s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......................X............................] duration: 1.84565807s FAIL (2025-07-28 00:50:24 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.845686525s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.X..........X.....X.X...........X.................] duration: 5.983021674s FAIL (2025-07-28 00:50:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.983049168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X.......X.XX....X....X.......X.....X..............] duration: 5.893372821s FAIL (2025-07-28 00:50:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.893405033s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X.......X.XX....X....X.......X.....X.............] duration: 5.881630863s FAIL (2025-07-28 00:49:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.881658005s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.242071577s FAIL (2025-07-28 00:49:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== loading piece... Jul 28 00:48:51.275021562 (85h13m0.997584593s) [noise=true , address=104.36.82.50:4027 , piece_id=PB36YXNUQHTCLZVH3X7HTS2NBRB4LJ5UY4TBE6VIDMUS2M3NMFNA, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.275457617 (85h13m0.998020657s) [noise=true , address=104.36.82.5:4847 , piece_id=LS2CPKSJ3OTPOBTWHKCAT6BHRA275YRLFEICJDU7RHHZ3JI2NY4A, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.275889523 (85h13m0.998452563s) [noise=true , address=79.127.220.97:4375 , piece_id=375JMDQVFPME7EOLUBD3HN435QSPDFH6FMCBSKVRB2UI5OC4I56Q, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.277349897 (85h13m0.999912937s) [noise=true , address=104.36.82.41:4764 , piece_id=IOISLNZCQBBT4L5PFG66Q6W4RKGGFMQ55C2GOLM55HR7LIO6QW3A, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.278765596 (85h13m1.001328636s) [noise=true , address=104.36.82.20:4470 , piece_id=5CXBC2UYZXPZOH3IP2M42EZPN4NE3RGXFDCB2FWPKAGTDA5RSAMA, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.279038779 (85h13m1.001601819s) [noise=true , address=79.127.201.215:4352 , piece_id=L76L53UAAWK2CTAKRXNZRWHSKVFZ6F4ZAEMIOWUO5XWMGXTKLMQQ, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.279338282 (85h13m1.001901312s) [noise=true , address=104.36.82.51:4069 , piece_id=SITV7IVTNJK3AUVADO67PVY66FZRO25RDEHQJUP2A7KQJABIYAZQ, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.279645289 (85h13m1.002208329s) [noise=true , address=104.36.82.11:4134 , piece_id=BAUSKQ25YDAFAYYP6QPU45JD4BWD5STZONYQJ454FBY57UOPXSPA, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.279881300 (85h13m1.00244434s) [noise=true , address=104.36.82.42:4812 , piece_id=7Q3UW6WFNSIHFDBGUXJTHSHBDQUFR5HHCYWP6CD4UT4ECVHIS6UQ, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:52.685134381 (85h13m2.407697422s) [upload=81766]: Next returned nil segment Jul 28 00:48:52.685155181 (85h13m2.407718222s) [upload=81766]: Waiting for error group managing segments... Jul 28 00:48:53.040284716 (85h13m2.762847756s) [noise=true , address=79.127.220.97:4375 , piece_id=375JMDQVFPME7EOLUBD3HN435QSPDFH6FMCBSKVRB2UI5OC4I56Q, seg_pos={0 0}, upload=81766]: Done uploading piece. err: <nil> Jul 28 00:48:53.206723041 (85h13m2.929286081s) [noise=true , address=104.36.82.5:4847 , piece_id=LS2CPKSJ3OTPOBTWHKCAT6BHRA275YRLFEICJDU7RHHZ3JI2NY4A, seg_pos={0 0}, upload=81766]: Done uploading piece. err: <nil> Jul 28 00:49:00.337488064 (85h13m10.060051104s) [noise=true , address=79.127.201.215:4352 , piece_id=L76L53UAAWK2CTAKRXNZRWHSKVFZ6F4ZAEMIOWUO5XWMGXTKLMQQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: <nil> Jul 28 00:49:50.175636429 (85h13m59.898199469s) [noise=true , address=104.36.82.13:4194 , piece_id=WXCHIB6HGFBZA73PDZ4ZZAE2IM5PYOFR75X4MPPMV3QVVLF7Z57Q, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:104.36.82.13:4194): protocol: rpc: dial tcp 104.36.82.13:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176027898 (85h13m59.898590938s) [noise=true , address=104.36.82.21:4494 , piece_id=4FXJTE44INMIQLKJ6KJ7WT2ZYKSST6QBM7SZUTFD5QK2PEWDJSTQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1sRaMuQG18B5SfPpzxp5VWjrwe38SS38g4askmbURxU9tepJHi, address:104.36.82.21:4494): protocol: rpc: dial tcp 104.36.82.21:4494: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176069497 (85h13m59.898632537s) [noise=true , address=104.36.82.4:4823 , piece_id=LME7UGRE24RARE3QR53LISSKLYZ5RJOHHX7LDUMRSKQMG6AGNLVQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1JWhB5QxbZsTD863tEfkEN1CUbKBMQvMJ9ZaiMAwKvxxXyDkEq, address:104.36.82.4:4823): protocol: rpc: dial tcp 104.36.82.4:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176203493 (85h13m59.898766543s) [noise=true , address=104.36.82.41:4764 , piece_id=IOISLNZCQBBT4L5PFG66Q6W4RKGGFMQ55C2GOLM55HR7LIO6QW3A, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176486344 (85h13m59.899049394s) [noise=true , address=104.36.82.11:4134 , piece_id=BAUSKQ25YDAFAYYP6QPU45JD4BWD5STZONYQJ454FBY57UOPXSPA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: write tcp 188.166.225.170:43398->104.36.82.11:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176519699 (85h13m59.899082739s) [noise=true , address=104.36.82.55:4201 , piece_id=FAS6SCHFP7OLFEZUUL45SCJX677SG2CEQNZNRMDPD74DTOHGQQ6Q, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176628407 (85h13m59.899191447s) [noise=true , address=104.36.82.52:4093 , piece_id=5KJHD2CH2M3ZAJXLQRZPHVGGZAFTDY3GNIQAK3CIDV4TD2KC5B2Q, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: write tcp 188.166.225.170:58120->104.36.82.52:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176712778 (85h13m59.899275818s) [noise=true , address=104.36.82.50:4027 , piece_id=PB36YXNUQHTCLZVH3X7HTS2NBRB4LJ5UY4TBE6VIDMUS2M3NMFNA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: rpc: dial tcp 104.36.82.50:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176780688 (85h13m59.899343728s) [noise=true , address=104.36.82.51:4069 , piece_id=SITV7IVTNJK3AUVADO67PVY66FZRO25RDEHQJUP2A7KQJABIYAZQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe, address:104.36.82.51:4069): protocol: write tcp 188.166.225.170:58972->104.36.82.51:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176818711 (85h13m59.899381751s) [noise=true , address=104.36.82.42:4812 , piece_id=7Q3UW6WFNSIHFDBGUXJTHSHBDQUFR5HHCYWP6CD4UT4ECVHIS6UQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1TmXeVTeejZvrWi2RPxMutM6kXuZiMCtPxQBwL4tq6mhoLwFQo, address:104.36.82.42:4812): protocol: write tcp 188.166.225.170:52326->104.36.82.42:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176848317 (85h13m59.899411358s) [noise=true , address=104.36.82.44:4896 , piece_id=MJ6NC2B52DJQULVKLOECW7A5G3CJ32CDQ7RIDCIU4I62POULOAYA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 188.166.225.170:37744->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176887001 (85h13m59.899450041s) [noise=true , address=104.36.82.54:4189 , piece_id=TQHXBE4YM6N5WGUPV56SHNSQNKM77AP3JSFNIXCZCLUGLEKHJPZA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: rpc: dial tcp 104.36.82.54:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176952496 (85h13m59.899515546s) [noise=true , address=104.36.82.53:4132 , piece_id=HJMWCQIBW7TGCWYPBHNEB6PICKLSYJAZCCOTWSDCX5VICDONS7YA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: write tcp 188.166.225.170:49304->104.36.82.53:4132: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176988045 (85h13m59.899551085s) [noise=true , address=104.36.82.20:4470 , piece_id=5CXBC2UYZXPZOH3IP2M42EZPN4NE3RGXFDCB2FWPKAGTDA5RSAMA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12ZbWCEiPuqJqMhz88MJZrw7ar8wrV5abTTagADXTpnU8NQ8zYH, address:104.36.82.20:4470): protocol: rpc: dial tcp 104.36.82.20:4470: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.177036597 (85h13m59.899599637s) [noise=true , address=104.36.82.56:4253 , piece_id=EE7SQW377XIL2O34AMC3P22OBPJOUSWJOAB5PE7DKGULFCOCV3LQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: write tcp 188.166.225.170:50860->104.36.82.56:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.177052147 (85h13m59.899615187s) [seg_pos={0 0}, upload=81766]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:49:50.415529002 (85h14m0.138092042s) [upload=81766]: Done uploading segments... Jul 28 00:49:50.415554239 (85h14m0.138117279s) [upload=81766]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.24208367s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)