Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [...........X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23043576s FAIL (2025-07-28 00:54:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== .. Jul 28 00:53:57.628167482 (85h18m7.350730522s) [noise=true , address=104.36.82.21:4508 , piece_id=L6QHYBRMETGENWZBIGB4W5USSDIRLE4TMS3YNXI5RXOATBSJSPXQ, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:57.629174760 (85h18m7.3517378s) [noise=true , address=104.36.82.2:4760 , piece_id=ZWZS32NBI5YANFACH5HK6BTYDLTD6B26HEXCNNMYVRNYW7QNB5PQ, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:57.629484382 (85h18m7.352047422s) [noise=true , address=104.36.82.14:4225 , piece_id=Z2ZUOUZJZ4XKILQCWSUPQBQCTIVZ4Q3DV5S5WVWSWHZFWSBHX3DA, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:57.629720824 (85h18m7.352283865s) [noise=true , address=104.36.82.12:4180 , piece_id=JA6Y5DTLIS34SLCIQHQYKNAHMA5F32XSJBNMNJVF7ZRH6LEQ3SBQ, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:57.630265556 (85h18m7.352828596s) [noise=true , address=104.36.82.15:4285 , piece_id=URB43AMIUIUUOZAM37SL42H5FUOFIMX7B7FLEB3TIA6E52QUCPXQ, seg_pos={0 0}, upload=81845]: Uploading piece... Jul 28 00:53:59.149248104 (85h18m8.871811134s) [upload=81845]: Next returned nil segment Jul 28 00:53:59.149266138 (85h18m8.871829178s) [upload=81845]: Waiting for error group managing segments... Jul 28 00:54:02.641709492 (85h18m12.364272532s) [noise=true , address=79.127.201.215:4347 , piece_id=2LIPMUCFKL2544IMN7AUC4DRFSIEWJCL7OSZMUT5IYJVC7OTP7GQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: <nil> Jul 28 00:54:56.515684334 (85h19m6.238247444s) [noise=true , address=104.36.82.16:4320 , piece_id=32OP2NA6YWLWZX2QFFYSGPSWNO33WP2GV7T3VRP247XH4UW25EIA, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B, address:104.36.82.16:4320): protocol: write tcp 188.166.225.170:32954->104.36.82.16:4320: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.515777432 (85h19m6.238340492s) [noise=true , address=104.36.82.10:4109 , piece_id=C4OFGLUGO7URQZYVQWBVL5FTMEHT7HE3ZKPSV7HQ7LIBOSYFIL5Q, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12ZE1ut9Tum4tzq2m8zbFaGUNLdNZSZQYARkayDAfE1b7uZtWjW, address:104.36.82.10:4109): protocol: write tcp 188.166.225.170:57884->104.36.82.10:4109: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.515862555 (85h19m6.238425645s) [noise=true , address=104.36.82.6:4905 , piece_id=XURLVQTPGGAJI4REOW7UELMFX5NV7AEKMQMTGSU3PDTENMO4ADWQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1qAL8paQ8Z3A9VgPRQ2UUG2LytmYZKdz9hjLzK8w8fQmHiYf88, address:104.36.82.6:4905): protocol: write tcp 188.166.225.170:47648->104.36.82.6:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.515938951 (85h19m6.238502081s) [noise=true , address=104.36.82.11:4117 , piece_id=4BO72GMCSVO32JFUQOPMNUASS6E5IS3QFIXNXHMKGOATDPARHQSQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the, address:104.36.82.11:4117): protocol: write tcp 188.166.225.170:44712->104.36.82.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516023292 (85h19m6.238586373s) [noise=true , address=104.36.82.41:4792 , piece_id=UHLYGW6R74SOLE5MFIHQL242RATJPYMUN2IL6IZBK4PAWB4VVY6Q, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: write tcp 188.166.225.170:36322->104.36.82.41:4792: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516085311 (85h19m6.238648341s) [noise=true , address=104.36.82.2:4760 , piece_id=ZWZS32NBI5YANFACH5HK6BTYDLTD6B26HEXCNNMYVRNYW7QNB5PQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:104.36.82.2:4760): protocol: write tcp 188.166.225.170:54414->104.36.82.2:4760: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516129977 (85h19m6.238693037s) [noise=true , address=104.36.82.15:4285 , piece_id=URB43AMIUIUUOZAM37SL42H5FUOFIMX7B7FLEB3TIA6E52QUCPXQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y, address:104.36.82.15:4285): protocol: write tcp 188.166.225.170:36972->104.36.82.15:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516138362 (85h19m6.238701412s) [noise=true , address=104.36.82.21:4508 , piece_id=L6QHYBRMETGENWZBIGB4W5USSDIRLE4TMS3YNXI5RXOATBSJSPXQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:104.36.82.21:4508): protocol: write tcp 188.166.225.170:56478->104.36.82.21:4508: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516173851 (85h19m6.238736891s) [noise=true , address=104.36.82.52:4112 , piece_id=NQGLXVN5VABGY5WEV62VGTCEQAHIA6KR6VWL657C33NU7M3XFYXA, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: write tcp 188.166.225.170:36752->104.36.82.52:4112: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516210361 (85h19m6.238773421s) [noise=true , address=104.36.82.22:4516 , piece_id=YPVSAKV3HE5JRBL664F2XY2CUFKAJNUXX5YJOECGAFJRHW64A24A, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:104.36.82.22:4516): protocol: write tcp 188.166.225.170:34646->104.36.82.22:4516: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516221221 (85h19m6.238784271s) [noise=true , address=104.36.82.55:4194 , piece_id=IK2HRKYBVPHBMXVXQOYG5O2BD5BBMTUPNVRSNVM3Z5QBOKBQDEQQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: write tcp 188.166.225.170:42578->104.36.82.55:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516263011 (85h19m6.238826071s) [noise=true , address=104.36.82.14:4225 , piece_id=Z2ZUOUZJZ4XKILQCWSUPQBQCTIVZ4Q3DV5S5WVWSWHZFWSBHX3DA, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:104.36.82.14:4225): protocol: write tcp 188.166.225.170:51994->104.36.82.14:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516306754 (85h19m6.238869795s) [noise=true , address=104.36.82.12:4180 , piece_id=JA6Y5DTLIS34SLCIQHQYKNAHMA5F32XSJBNMNJVF7ZRH6LEQ3SBQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: write tcp 188.166.225.170:44012->104.36.82.12:4180: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516330119 (85h19m6.238893159s) [noise=true , address=104.36.82.13:4219 , piece_id=S357VGA6B3ZQ2TD25THRYAWYV4LORN3IG2AW3M54SCJRIN2UDZ7A, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12fUzLg93yKvyKkNvuvcKZHyvJ2HGE6tUCg1Wm7ZZjKKeDZDWm9, address:104.36.82.13:4219): protocol: write tcp 188.166.225.170:44550->104.36.82.13:4219: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516372701 (85h19m6.238935731s) [noise=true , address=104.36.82.22:4533 , piece_id=QOTOPEX2CQXCARA2B7X3UBN2PEZBXG2A22AXN3766VONJMA2HOSQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12LXK5pbXavbPZh4YSuEkgk7wsRqwvcVEHv4w9FDRYqSkEU89VF, address:104.36.82.22:4533): protocol: write tcp 188.166.225.170:50010->104.36.82.22:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516383341 (85h19m6.238946381s) [noise=true , address=104.36.82.51:4078 , piece_id=4AEEVMRRMBVSEDKW6FJQ4LGDOX36MFP2UGFNHD4VGXOKPTWIUATQ, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd, address:104.36.82.51:4078): protocol: write tcp 188.166.225.170:46132->104.36.82.51:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516389613 (85h19m6.238952653s) [noise=true , address=104.36.82.47:4528 , piece_id=MUWLZWYALT6OUZK4P3TGAWDKI6SFY5V3MQJHVES5XNTLRIG5HN5Q, seg_pos={0 0}, upload=81845]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: write tcp 188.166.225.170:43538->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:54:56.516446251 (85h19m6.239009281s) [seg_pos={0 0}, upload=81845]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:54:56.744110367 (85h19m6.466673397s) [upload=81845]: Done uploading segments... Jul 28 00:54:56.744135285 (85h19m6.466698325s) [upload=81845]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.230508429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X..........X.....X.X...........X...................X...] duration: 5.822676939s FAIL (2025-07-28 00:54:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.822703959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.......X.XX....X....X.......X.....X....................] duration: 5.725334524s FAIL (2025-07-28 00:54:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.725360955s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................X.................................] duration: 1.891316039s FAIL (2025-07-28 00:53:57 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.891339043s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [............X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235751492s FAIL (2025-07-28 00:53:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ading piece... Jul 28 00:52:57.156942030 (85h17m6.87950507s) [noise=true , address=104.36.82.5:4864 , piece_id=2FR6IDHUOG67JLVOCGBSHMFFPZJI77ECI45ZY6FRKGEV32BZSRTQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157071438 (85h17m6.879634468s) [noise=true , address=104.36.82.15:4281 , piece_id=VM3KYKNERRFAUI5ZXDLMRLSVFG37UPCV5KYKEM63JP6UURP5X44Q, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157151572 (85h17m6.879714612s) [noise=true , address=79.127.201.215:4347 , piece_id=6GUNVFTJFOB4NCFUQJ3AN7WQBODY236O3LGQFUSXY7PG36UPKZTQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157405357 (85h17m6.879968387s) [noise=true , address=104.36.82.39:4735 , piece_id=AM74U7EJ5CF3YYHPM7TIB4XLGLV7VLQOSAX72XR4CRWP5CO7OKDA, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157416599 (85h17m6.879979639s) [noise=true , address=104.36.82.13:4203 , piece_id=EU7Z7XATZTTX2DCE7KFSK5WBAIWJCBB5SYLZ4HKWZ5FF2CM5RRFQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.157833877 (85h17m6.880396907s) [noise=true , address=104.36.82.21:4503 , piece_id=WQUEBRVA75MWPVFVIVYM5ZNEKS6GPOJYLPCWKMDMQ2JNOVCKJ4BQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.158381635 (85h17m6.880944675s) [noise=true , address=104.36.82.20:4454 , piece_id=LIKII3N3JAF2F6SRGVPQ6BPHDG64B5QZPOLBCFO3CMF7T3DHW6AQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.158865310 (85h17m6.88142835s) [noise=true , address=104.36.82.11:4145 , piece_id=YJ3F7JXTTVRYCZPXLM7YLSHFRTNHOM7TSYXTTU4TY4OLFE6LPMWQ, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:57.159322155 (85h17m6.881885195s) [noise=true , address=104.36.82.47:4517 , piece_id=M74A7A26ARCRGASU626SHFJEYAUZ2PHOPRMVLTSXWFG7ZRFT4UPA, seg_pos={0 0}, upload=81829]: Uploading piece... Jul 28 00:52:58.618677399 (85h17m8.341240439s) [upload=81829]: Next returned nil segment Jul 28 00:52:58.618699211 (85h17m8.341262251s) [upload=81829]: Waiting for error group managing segments... Jul 28 00:53:01.801272985 (85h17m11.523836045s) [noise=true , address=79.127.201.215:4347 , piece_id=6GUNVFTJFOB4NCFUQJ3AN7WQBODY236O3LGQFUSXY7PG36UPKZTQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: <nil> Jul 28 00:53:03.216039458 (85h17m12.938602498s) [noise=true , address=64.49.13.140:4509 , piece_id=GTWJTOVM6PRC6F3I26DPWCG5SUNO3C2QSF2NHGO4YCVAHCKBEICQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: <nil> Jul 28 00:53:56.050333153 (85h18m5.772896193s) [noise=true , address=104.36.82.45:4920 , piece_id=PHTGUIMR3UXAVH7OA6LUARLUZRP35BB53SDS3DKFY5LCNSRFWQZA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S, address:104.36.82.45:4920): protocol: rpc: dial tcp 104.36.82.45:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050374954 (85h18m5.772937994s) [noise=true , address=104.36.82.56:4242 , piece_id=7WHETCKOD7LFXFQMECEJYULOFQRWAXNCTUJBD3LOV45Z65LVN7FA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: rpc: dial tcp 104.36.82.56:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050404460 (85h18m5.7729675s) [noise=true , address=104.36.82.12:4159 , piece_id=SJKCI2LWPEC5CMPF23HWM4BUTTRKKPJRC2UITNVJUUL3HB5VSIIQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: rpc: dial tcp 104.36.82.12:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050433325 (85h18m5.772996365s) [noise=true , address=104.36.82.13:4203 , piece_id=EU7Z7XATZTTX2DCE7KFSK5WBAIWJCBB5SYLZ4HKWZ5FF2CM5RRFQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:18cNSsgqn3n6z5LxRcVCRzDzfFDpvptV1MKh6NdNRLW4bg9fWM, address:104.36.82.13:4203): protocol: rpc: dial tcp 104.36.82.13:4203: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050459796 (85h18m5.773022836s) [noise=true , address=104.36.82.21:4503 , piece_id=WQUEBRVA75MWPVFVIVYM5ZNEKS6GPOJYLPCWKMDMQ2JNOVCKJ4BQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:17F3Gn5LYgekp8WgagEdFXtRu8jCc27qA9UuW7pWsJ9FUFKFSi, address:104.36.82.21:4503): protocol: rpc: dial tcp 104.36.82.21:4503: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050509121 (85h18m5.773072161s) [noise=true , address=104.36.82.53:4154 , piece_id=NDUJHIYDPNNFGQRCYOY4PJHPBTME6YZZSFYSD5J3FEMMAZZO224A, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050550409 (85h18m5.773113449s) [noise=true , address=104.36.82.39:4735 , piece_id=AM74U7EJ5CF3YYHPM7TIB4XLGLV7VLQOSAX72XR4CRWP5CO7OKDA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1QCQZ8eFGLa5TAaAJJzUip8o5DAELzTBTm4UFbt3Am5yYWXAZF, address:104.36.82.39:4735): protocol: rpc: dial tcp 104.36.82.39:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050578974 (85h18m5.773142004s) [noise=true , address=104.36.82.20:4454 , piece_id=LIKII3N3JAF2F6SRGVPQ6BPHDG64B5QZPOLBCFO3CMF7T3DHW6AQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:104.36.82.20:4454): protocol: rpc: dial tcp 104.36.82.20:4454: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050609382 (85h18m5.773172422s) [noise=true , address=104.36.82.47:4517 , piece_id=M74A7A26ARCRGASU626SHFJEYAUZ2PHOPRMVLTSXWFG7ZRFT4UPA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050639088 (85h18m5.773202129s) [noise=true , address=104.36.82.55:4228 , piece_id=KQPAJIBJRSUOTHLU7B4LXGP7HJWLQ5DWA3XIYOCNAMZ4ECXDV2OQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: rpc: dial tcp 104.36.82.55:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050665539 (85h18m5.773228589s) [noise=true , address=104.36.82.11:4145 , piece_id=YJ3F7JXTTVRYCZPXLM7YLSHFRTNHOM7TSYXTTU4TY4OLFE6LPMWQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: rpc: dial tcp 104.36.82.11:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050679336 (85h18m5.773242376s) [noise=true , address=104.36.82.15:4281 , piece_id=VM3KYKNERRFAUI5ZXDLMRLSVFG37UPCV5KYKEM63JP6UURP5X44Q, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2, address:104.36.82.15:4281): protocol: rpc: dial tcp 104.36.82.15:4281: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050695446 (85h18m5.773258486s) [noise=true , address=104.36.82.14:4241 , piece_id=F3LNXFISZEFQMYJPMOPHB5WSTO2H5NUGUAT2KKZKYZSBG6RDG3EA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12aBy6nQScbjzmBSScfEZwTX8NCcP9c71FRQkUH54WvZhhhtXLR, address:104.36.82.14:4241): protocol: rpc: dial tcp 104.36.82.14:4241: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050711217 (85h18m5.773274257s) [noise=true , address=104.36.82.7:4943 , piece_id=EA7LC6TR7VJME2LQIBJGQNYDGOTHDYBD2GECX5KGPWSVW2PSYRDA, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12BdCdPHn6ko7nfY4sPuc4NutXABzJDdHHhtSnyXri5gofhZEZJ, address:104.36.82.7:4943): protocol: write tcp 188.166.225.170:39156->104.36.82.7:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050722879 (85h18m5.773285919s) [noise=true , address=104.36.82.49:4999 , piece_id=PGUGUPCSP2PWCHXWZNZGRX7T3XFOWOF7DTKPBTMHFKE3WVTLO7AQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:1tSiyPFy1xn1YRCLmKmPmbTm3LUmDP9nT1YRCXnsuyY6XsLKrG, address:104.36.82.49:4999): protocol: rpc: dial tcp 104.36.82.49:4999: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050741775 (85h18m5.773304805s) [noise=true , address=104.36.82.5:4864 , piece_id=2FR6IDHUOG67JLVOCGBSHMFFPZJI77ECI45ZY6FRKGEV32BZSRTQ, seg_pos={0 0}, upload=81829]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: write tcp 188.166.225.170:39984->104.36.82.5:4864: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:53:56.050759730 (85h18m5.77332277s) [seg_pos={0 0}, upload=81829]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:53:56.284294548 (85h18m6.006857588s) [upload=81829]: Done uploading segments... Jul 28 00:53:56.284320297 (85h18m6.006883327s) [upload=81829]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.235767333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X..........X.....X.X...........X...................X..] duration: 5.871791097s FAIL (2025-07-28 00:53:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.871818689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X.......X.XX....X....X.......X.....X...................] duration: 5.76684137s FAIL (2025-07-28 00:53:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.766868079s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................X................................] duration: 1.839691876s FAIL (2025-07-28 00:53:28 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.839717235s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.......X.XX....X....X.......X.....X..................] duration: 5.806881307s FAIL (2025-07-28 00:53:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.806910833s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.X..........X.....X.X...........X...................X.] duration: 5.770423552s FAIL (2025-07-28 00:52:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.770441186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [.............X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238488656s FAIL (2025-07-28 00:52:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 0:51:56.628136725 (85h16m6.350699765s) [noise=true , address=104.36.82.15:4269 , piece_id=6AUB6RHUX75GUGKGWCG4DOJGOALIJIUPRP6IGHA5QUV2TGBUNFFA, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.628529166 (85h16m6.351092206s) [noise=true , address=104.36.82.52:4088 , piece_id=NTM2SUYNUUCGGRWJGEJBZZSYFNV77ZPNL45QVMEL7VF3EZG6OULQ, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.628785516 (85h16m6.351348546s) [noise=true , address=104.36.82.12:4161 , piece_id=W2GXL43MFJRVHISR3BG3GUE37ESM324WZ4BETSXEFHIQUMS7NM4A, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.628990168 (85h16m6.351553198s) [noise=true , address=104.36.82.13:4205 , piece_id=XWOXWEZYH7DKMK3L7HN6BDICKZUID3NC2BCX2CHSI6JCLVV6O32A, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.629227502 (85h16m6.351790542s) [noise=true , address=104.36.82.53:4147 , piece_id=AMLUVXYQ6TQGLC2MZZSKCNQ5GSR6TL35ASZTZ64RQF3X5NMJM3GQ, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.629487329 (85h16m6.352050369s) [noise=true , address=104.36.82.56:4264 , piece_id=MJZCE23A5F6ILUNLNYSWHB3TXYKCAAQGZ7J372NT3V7RMCRBPYEQ, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.629683274 (85h16m6.352246304s) [noise=true , address=104.36.82.44:4894 , piece_id=QM3BMURZGNFIFLEGFZE5UH5TBQE6HQGI32YSU7VNATR7DHMZZI3Q, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:56.629939534 (85h16m6.352502574s) [noise=true , address=64.49.13.137:4498 , piece_id=AONBCU6MLEV64NEFBACPQTKA57IPA4GE35K32U4O56X655PDVKTA, seg_pos={0 0}, upload=81816]: Uploading piece... Jul 28 00:51:57.751189877 (85h16m7.473752917s) [upload=81816]: Next returned nil segment Jul 28 00:51:57.751201469 (85h16m7.473764509s) [upload=81816]: Waiting for error group managing segments... Jul 28 00:51:58.033568719 (85h16m7.756131759s) [noise=true , address=64.49.13.137:4498 , piece_id=AONBCU6MLEV64NEFBACPQTKA57IPA4GE35K32U4O56X655PDVKTA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: <nil> Jul 28 00:52:55.498735625 (85h17m5.221298665s) [noise=true , address=104.36.82.21:4501 , piece_id=QSOSARYHH577ULBMYT3RHSQYCC4VKTLZ5GFOFG5VHQQGOSHDUFIA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1e1R3gjh5CYJdDqck7xcR6A19Hb443FWbX8X2AQikxt5f4AVvU, address:104.36.82.21:4501): protocol: rpc: dial tcp 104.36.82.21:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498781994 (85h17m5.221345034s) [noise=true , address=104.36.82.55:4223 , piece_id=UIGTXCPQHX6N37PDDYLYSR3BJJIU3VQEXBX5E4NIVFOHNQB6TWVQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: rpc: dial tcp 104.36.82.55:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498818143 (85h17m5.221381173s) [noise=true , address=104.36.82.55:4211 , piece_id=BQ7B5C4W5743MQUY5SMTUAFLDKGWIQI4JPFVHQTZICPZTAQIMM3Q, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: rpc: dial tcp 104.36.82.55:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498831679 (85h17m5.221394719s) [noise=true , address=104.36.82.47:4534 , piece_id=XX4Y5GWLLITF2YKN2QBMK2FRGJ47ONITXTJEDFA7QDJF4JCXXLQA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: rpc: dial tcp 104.36.82.47:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498849933 (85h17m5.221412973s) [noise=true , address=104.36.82.10:4088 , piece_id=NWJRJAN5HYHAZ234NMP2OEFQ34UUJA3MZVQEKSKHKNACHBQLGN3Q, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, address:104.36.82.10:4088): protocol: rpc: dial tcp 104.36.82.10:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498886244 (85h17m5.221449274s) [noise=true , address=104.36.82.4:4824 , piece_id=E5YUSGVBFZ6A7X2D3AM26PA6HI27KDFB56444PETMNMUM7PDJQDA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498893026 (85h17m5.221456066s) [noise=true , address=104.36.82.14:4244 , piece_id=2A7OABT2E3EUQXFOWXBRHP3X5BCHV2JPVJOTZRJFIMA7PTOSIFVA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:4244): protocol: rpc: dial tcp 104.36.82.14:4244: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498924586 (85h17m5.221487627s) [noise=true , address=104.36.82.2:4762 , piece_id=BC7J47MJNZELCKRHDXJL3F6YVOGIHZCCNF7O7R4ZCMMCMA6J5CAA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498935878 (85h17m5.221498918s) [noise=true , address=104.36.82.11:4143 , piece_id=DBP7SN5QO7USRS4X6OCGETT72UZGWQZ66K4LFFAKFPQOZLCR5MUA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: rpc: dial tcp 104.36.82.11:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498960625 (85h17m5.221523665s) [noise=true , address=104.36.82.56:4264 , piece_id=MJZCE23A5F6ILUNLNYSWHB3TXYKCAAQGZ7J372NT3V7RMCRBPYEQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498979241 (85h17m5.221542281s) [noise=true , address=104.36.82.52:4088 , piece_id=NTM2SUYNUUCGGRWJGEJBZZSYFNV77ZPNL45QVMEL7VF3EZG6OULQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:104.36.82.52:4088): protocol: rpc: dial tcp 104.36.82.52:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.498992376 (85h17m5.221555416s) [noise=true , address=104.36.82.44:4894 , piece_id=QM3BMURZGNFIFLEGFZE5UH5TBQE6HQGI32YSU7VNATR7DHMZZI3Q, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499017654 (85h17m5.221580694s) [noise=true , address=104.36.82.12:4161 , piece_id=W2GXL43MFJRVHISR3BG3GUE37ESM324WZ4BETSXEFHIQUMS7NM4A, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:1SdSwg6tZCKSturNytQmWUmSWQ2G5dUSrb2RA8hxLLpBtBE6tf, address:104.36.82.12:4161): protocol: rpc: dial tcp 104.36.82.12:4161: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499024338 (85h17m5.221587378s) [noise=true , address=104.36.82.54:4165 , piece_id=HYLWQ4OJ2RI5H7V7IHMLOAF6L22BMOPHVJ77N2EBSCEO3PAI4JYQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499059004 (85h17m5.221622044s) [noise=true , address=104.36.82.15:4269 , piece_id=6AUB6RHUX75GUGKGWCG4DOJGOALIJIUPRP6IGHA5QUV2TGBUNFFA, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12V8v5nzyyouSqndATi4fRJUC6HgFbhguBjnmwpki48zdRBTcgC, address:104.36.82.15:4269): protocol: rpc: dial tcp 104.36.82.15:4269: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499069994 (85h17m5.221633034s) [noise=true , address=104.36.82.13:4205 , piece_id=XWOXWEZYH7DKMK3L7HN6BDICKZUID3NC2BCX2CHSI6JCLVV6O32A, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr, address:104.36.82.13:4205): protocol: write tcp 188.166.225.170:52366->104.36.82.13:4205: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499107737 (85h17m5.221670777s) [noise=true , address=104.36.82.53:4147 , piece_id=AMLUVXYQ6TQGLC2MZZSKCNQ5GSR6TL35ASZTZ64RQF3X5NMJM3GQ, seg_pos={0 0}, upload=81816]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 188.166.225.170:52012->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:55.499135220 (85h17m5.22169826s) [seg_pos={0 0}, upload=81816]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:52:55.734778493 (85h17m5.457341533s) [upload=81816]: Done uploading segments... Jul 28 00:52:55.734803110 (85h17m5.45736614s) [upload=81816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.238514675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.X..........X.....X.X...........X...................X] duration: 30.232625465s FAIL (2025-07-28 00:52:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232646775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X.......................X...............................] duration: 1.820323489s FAIL (2025-07-28 00:52:46 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.820348719s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.......X.XX....X....X.......X.....X.................] duration: 5.825094181s FAIL (2025-07-28 00:52:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.82511987s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [..............X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230475841s FAIL (2025-07-28 00:51:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== , piece_id=FM6XCLSEDVSO6D7IQ66PLBS4Q2KCX6JYHYD3L377OYWROWSKKGHQ, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.671947809 (85h15m2.394510849s) [noise=true , address=104.36.82.6:4899 , piece_id=X4ZRUYPCSO3HO4AITFBYCAKBMRFDSGWFQVO4ONB2RRZGT2MUAR7A, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.672152380 (85h15m2.39471542s) [noise=true , address=104.36.82.21:4507 , piece_id=4KKEMFBCU5FVZ4LFXAU4E6PTE3DSPNUXG3GZYEPVGWJ6MGPBNMGA, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.672513801 (85h15m2.395076841s) [noise=true , address=104.36.82.2:4740 , piece_id=5SSP27FUANOLHIYTFJDDWLAK2KJ7GKW5A36WYRWTFERJIQDLVFUQ, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.672769250 (85h15m2.39533229s) [noise=true , address=104.36.82.12:4176 , piece_id=XSY5C7OJJXYLU7W2OTEGPOEVTJGIH2UVJQQ2SAL3ZGLUKSNONF7A, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.672962099 (85h15m2.395525139s) [noise=true , address=104.36.82.49:4001 , piece_id=TGTXQCFPUR4CKR4BMUM47M3WBCMWVPOTVC4EMBQCIAF6MWW4HTGQ, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:52.673148186 (85h15m2.395711226s) [noise=true , address=104.36.82.55:4199 , piece_id=6I7E4GUWCHEIDPKHP34CJPIW5KJ3WMPZMZHDHL7TC6QASBJJH5YQ, seg_pos={0 0}, upload=81796]: Uploading piece... Jul 28 00:50:53.779601427 (85h15m3.502164467s) [upload=81796]: Next returned nil segment Jul 28 00:50:53.779618670 (85h15m3.50218173s) [upload=81796]: Waiting for error group managing segments... Jul 28 00:50:54.091618936 (85h15m3.814182016s) [noise=true , address=79.127.205.235:4673 , piece_id=HGYTP5HVHLUOLIM6HTBNAHVT4XAJJG4A3SM7ZKJ7DRZK3GTVJPFA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: <nil> Jul 28 00:50:56.527577110 (85h15m6.25014015s) [noise=true , address=79.127.201.215:4356 , piece_id=ANJOP5GGNX3YB3WBFUHXLMPXKLNXZRLRVKC6FKHAYMJW5CC253WA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: <nil> Jul 28 00:51:51.558091026 (85h16m1.280654066s) [noise=true , address=104.36.82.15:4280 , piece_id=FM6XCLSEDVSO6D7IQ66PLBS4Q2KCX6JYHYD3L377OYWROWSKKGHQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1t4hMpMGCu1YMmNLX6fssTjWy9Xi5Fp8kdHTDmkhxH9qsZiYM6, address:104.36.82.15:4280): protocol: write tcp 188.166.225.170:41194->104.36.82.15:4280: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558135882 (85h16m1.280698922s) [noise=true , address=104.36.82.21:4507 , piece_id=4KKEMFBCU5FVZ4LFXAU4E6PTE3DSPNUXG3GZYEPVGWJ6MGPBNMGA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: write tcp 188.166.225.170:55388->104.36.82.21:4507: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558178855 (85h16m1.280741895s) [noise=true , address=104.36.82.12:4176 , piece_id=XSY5C7OJJXYLU7W2OTEGPOEVTJGIH2UVJQQ2SAL3ZGLUKSNONF7A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m, address:104.36.82.12:4176): protocol: rpc: dial tcp 104.36.82.12:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558207449 (85h16m1.280770489s) [noise=true , address=104.36.82.53:4152 , piece_id=IE7OPN62IUI2RKWI7ZVZFMRTG3KP2FARILU7ZV3LJAG65QDLS36Q, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 188.166.225.170:43414->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558237066 (85h16m1.280800096s) [noise=true , address=104.36.82.5:4860 , piece_id=VI6HIYB47YZ2JXCG2UIE3RFX3K2NQFWG74CSVCIWOPMH67JKRM4A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: write tcp 188.166.225.170:37850->104.36.82.5:4860: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558294115 (85h16m1.280857155s) [noise=true , address=104.36.82.49:4001 , piece_id=TGTXQCFPUR4CKR4BMUM47M3WBCMWVPOTVC4EMBQCIAF6MWW4HTGQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1iZnfkAxoot7KjtjYbxbSFvDhDFcLd8M63Qkh67GuMXMJjdruD, address:104.36.82.49:4001): protocol: write tcp 188.166.225.170:46144->104.36.82.49:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558349371 (85h16m1.280912421s) [noise=true , address=104.36.82.52:4086 , piece_id=CSC745NAKWCFE3GI66GZM3FAT734HQMMACRSHXTMP3BCTQZYV2HA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:104.36.82.52:4086): protocol: write tcp 188.166.225.170:44190->104.36.82.52:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558408944 (85h16m1.280971994s) [noise=true , address=104.36.82.11:4142 , piece_id=7J35VV7ENERICAC4GBD7G2GVTSFX56GTIOPGMGX5V3HIHI26IBKQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: write tcp 188.166.225.170:39942->104.36.82.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558454442 (85h16m1.281017482s) [noise=true , address=104.36.82.54:4190 , piece_id=MPRMQJYNI24CLMP5J3DNEYNQZWHUF67ERGX5L5YB5XT4Q2IWIWHQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: write tcp 188.166.225.170:39452->104.36.82.54:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558461024 (85h16m1.281024064s) [noise=true , address=104.36.82.6:4899 , piece_id=X4ZRUYPCSO3HO4AITFBYCAKBMRFDSGWFQVO4ONB2RRZGT2MUAR7A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 188.166.225.170:48118->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558485832 (85h16m1.281048872s) [noise=true , address=104.36.82.2:4740 , piece_id=5SSP27FUANOLHIYTFJDDWLAK2KJ7GKW5A36WYRWTFERJIQDLVFUQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:12wgQDbE17gP8c6kbKnNyywSkY8Hk7mchcMaymLBQzPwRsSB8QE, address:104.36.82.2:4740): protocol: write tcp 188.166.225.170:39742->104.36.82.2:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558512863 (85h16m1.281075903s) [noise=true , address=104.36.82.54:4158 , piece_id=3SW3QX7W3UPFHAS5WEW55TDKDQHFA3E2WS37NY3ATNA4CGDALXOA, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: write tcp 188.166.225.170:49416->104.36.82.54:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558516911 (85h16m1.281079941s) [noise=true , address=104.36.82.55:4199 , piece_id=6I7E4GUWCHEIDPKHP34CJPIW5KJ3WMPZMZHDHL7TC6QASBJJH5YQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: write tcp 188.166.225.170:35812->104.36.82.55:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558543111 (85h16m1.281106151s) [noise=true , address=104.36.82.3:4793 , piece_id=TCF77K4ZK3AILD5PNLUXJHI6AAHRM2S4BR6FG3NV7W37ZLJZTR2A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:104.36.82.3:4793): protocol: write tcp 188.166.225.170:41906->104.36.82.3:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558550255 (85h16m1.281113295s) [noise=true , address=104.36.82.56:4231 , piece_id=PXD4O2OMKKUVPDHLVZYYZPFIC5QA44IAFFA7KVLLNBKQLYTIW34A, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: write tcp 188.166.225.170:51018->104.36.82.56:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558562058 (85h16m1.281125098s) [noise=true , address=104.36.82.22:4528 , piece_id=7V3ISUWDHTHQXRQ4G3ZQYNURM36YVYAH6Q3ZBJ2QUVNPIW5XJKEQ, seg_pos={0 0}, upload=81796]: Done uploading piece. err: ecclient: upload failed (node:1nDzCSeAbqSKpZUVGcu4NEj32jsoqojqSpzfpKYoXciXEjsTwo, address:104.36.82.22:4528): protocol: write tcp 188.166.225.170:53542->104.36.82.22:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:51:51.558590011 (85h16m1.281153051s) [seg_pos={0 0}, upload=81796]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:51:51.786989094 (85h16m1.509552134s) [upload=81796]: Done uploading segments... Jul 28 00:51:51.787013430 (85h16m1.50957647s) [upload=81796]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.230487643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......................X..............................] duration: 1.839018387s FAIL (2025-07-28 00:51:50 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.839047593s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.X..........X.....X.X...........X...................] duration: 5.849208558s FAIL (2025-07-28 00:51:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.849234919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X.......X.XX....X....X.......X.....X................] duration: 5.71676098s FAIL (2025-07-28 00:51:03 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.716789935s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X.......................X.............................] duration: 1.927016072s FAIL (2025-07-28 00:50:56 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.927091887s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [...............X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.243951942s FAIL (2025-07-28 00:50:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 7877s) [noise=true , address=104.36.82.10:4111 , piece_id=XZXIDRKLRL4IHNHGFW2ZCZKIHXKUWW4EF33XSB7KVYP7AN6J3YBA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190002167 (85h14m1.912565207s) [noise=true , address=104.36.82.9:4019 , piece_id=B7T35DNTWGAM4QRETBQFCGQZ2AWTDFDRBJBIG5YFNGGB5ROG7QBQ, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190237006 (85h14m1.912800046s) [noise=true , address=79.127.201.215:4355 , piece_id=HTFDKRMIH6HT25D2VNKNSLFRNGMU23FZTXUNXBD5LMOU4G63I55Q, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190448781 (85h14m1.913011821s) [noise=true , address=104.36.82.56:4237 , piece_id=BYISKHFOSF5TURCNAUDETJAOBDD2OMBFKFJBSTI52CSGA7YOBMFA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190674222 (85h14m1.913237262s) [noise=true , address=104.36.82.52:4101 , piece_id=MBZ6GPSY4CYARF3WSURA64DTMQT5G6VO6CKZJZTOEHGSEFU6JWOA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.190924982 (85h14m1.913488012s) [noise=true , address=104.36.82.13:4216 , piece_id=KCL3AZJWRSP5HNPWVPO5EWPUPTU7UFWNUFIIYFIVNJ3KX6XYOPZA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.191166905 (85h14m1.913729945s) [noise=true , address=104.36.82.54:4180 , piece_id=LONM6HAJLGZMKFPXWVHYM56VYCEGGGHNYYA3SR2KIGD2UACQ2KAQ, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:52.191382778 (85h14m1.913945818s) [noise=true , address=104.36.82.50:4024 , piece_id=QFIOLHBIXB55PNLXMXCPH5TRWI4NYT7PNOFWS4E5XZ7LNY5SU7OA, seg_pos={0 0}, upload=81779]: Uploading piece... Jul 28 00:49:53.686581090 (85h14m3.40914413s) [upload=81779]: Next returned nil segment Jul 28 00:49:53.686598343 (85h14m3.409161383s) [upload=81779]: Waiting for error group managing segments... Jul 28 00:50:00.379103601 (85h14m10.101666641s) [noise=true , address=79.127.201.215:4355 , piece_id=HTFDKRMIH6HT25D2VNKNSLFRNGMU23FZTXUNXBD5LMOU4G63I55Q, seg_pos={0 0}, upload=81779]: Done uploading piece. err: <nil> Jul 28 00:50:51.087461893 (85h15m0.810024933s) [noise=true , address=104.36.82.10:4111 , piece_id=XZXIDRKLRL4IHNHGFW2ZCZKIHXKUWW4EF33XSB7KVYP7AN6J3YBA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:128oYxFebZ8HAoUNDf2Tmqi23Q1kuo2XgzWxKVRCGwaiuZBWtAw, address:104.36.82.10:4111): protocol: rpc: dial tcp 104.36.82.10:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087499605 (85h15m0.810062645s) [noise=true , address=104.36.82.46:4486 , piece_id=JW4K6HTRSUKZZOPH3PR5ATWJYXQT745VYM42D2JW4WP74OVGPGHA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic, address:104.36.82.46:4486): protocol: rpc: dial tcp 104.36.82.46:4486: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087517470 (85h15m0.81008051s) [noise=true , address=104.36.82.56:4237 , piece_id=BYISKHFOSF5TURCNAUDETJAOBDD2OMBFKFJBSTI52CSGA7YOBMFA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: rpc: dial tcp 104.36.82.56:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087535483 (85h15m0.810098524s) [noise=true , address=104.36.82.13:4216 , piece_id=KCL3AZJWRSP5HNPWVPO5EWPUPTU7UFWNUFIIYFIVNJ3KX6XYOPZA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu, address:104.36.82.13:4216): protocol: rpc: dial tcp 104.36.82.13:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087556945 (85h15m0.810119985s) [noise=true , address=104.36.82.2:4756 , piece_id=5DFDMVPOA6L6ZYNXW7LX7PZCMVOI4MMFDPTIA6KXAIBXJ6H6J56A, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1Pow7VVb1PRDBvHjEcS4bfnh3276S313jiGawszkhaJ582pDqQ, address:104.36.82.2:4756): protocol: write tcp 188.166.225.170:35166->104.36.82.2:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087569339 (85h15m0.810132369s) [noise=true , address=104.36.82.53:4154 , piece_id=JUUQPR3EVF7F2U4PJCAN5CFEYE5GQS4WRVCR46INWG2XIR5WTU6A, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087602733 (85h15m0.810165773s) [noise=true , address=104.36.82.54:4180 , piece_id=LONM6HAJLGZMKFPXWVHYM56VYCEGGGHNYYA3SR2KIGD2UACQ2KAQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087633902 (85h15m0.810196942s) [noise=true , address=104.36.82.15:4284 , piece_id=MIGIVE2BV4ZMSAHM77HXQY2BKNROJO6JPY7EFVFPBT4FTRSTAF2A, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1YAnLu8onh19Mx524VikQzCHEFW4U8dgtuLx8amtQAiReSCisF, address:104.36.82.15:4284): protocol: rpc: dial tcp 104.36.82.15:4284: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087659461 (85h15m0.810222511s) [noise=true , address=104.36.82.11:4124 , piece_id=AXSTMSP4XLXO23DC6VXBEH3RD67EABD64ZY353MHIZRAAW3MACYA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1QAgfinEkxYwjzvtr53FRritkoDNbfYAw21qcJuj3uCY4JT4cZ, address:104.36.82.11:4124): protocol: rpc: dial tcp 104.36.82.11:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087684840 (85h15m0.81024788s) [noise=true , address=104.36.82.50:4024 , piece_id=QFIOLHBIXB55PNLXMXCPH5TRWI4NYT7PNOFWS4E5XZ7LNY5SU7OA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1EAeBfX3Xyp8pBDUoS7VhyXwpNbXooSYgxZPxP9kEXT1rZwkoG, address:104.36.82.50:4024): protocol: rpc: dial tcp 104.36.82.50:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087692154 (85h15m0.810255194s) [noise=true , address=104.36.82.9:4019 , piece_id=B7T35DNTWGAM4QRETBQFCGQZ2AWTDFDRBJBIG5YFNGGB5ROG7QBQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:12JuQKLQnXiEcyCj6xthEXSPy2Wu5T1B9m4q9YteoK69rMqVydD, address:104.36.82.9:4019): protocol: rpc: dial tcp 104.36.82.9:4019: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087714757 (85h15m0.810277797s) [noise=true , address=104.36.82.12:4153 , piece_id=FN4ERZAEW44CMJJ4XPDT3IW35Z4VNK5LJPUVHVLRX6CNITCJ5NRQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:124myE1wFhvjA5paJy5tr2hRe8jZCoEdbJLdBwvss1ZnT4GvtZv, address:104.36.82.12:4153): protocol: rpc: dial tcp 104.36.82.12:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087721740 (85h15m0.81028478s) [noise=true , address=104.36.82.52:4101 , piece_id=MBZ6GPSY4CYARF3WSURA64DTMQT5G6VO6CKZJZTOEHGSEFU6JWOA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 188.166.225.170:57606->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087727952 (85h15m0.810290982s) [noise=true , address=104.36.82.41:4768 , piece_id=OONZCR7MTTKBAXGBBCTEFWCOWHQRSDT2DNDMMR2TIHOF5AKYDRWA, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087735858 (85h15m0.810298898s) [noise=true , address=104.36.82.3:4792 , piece_id=4RN7MWU5AFIDHHLE4773C4AGETWZWLMNEDTR3VCJG4SBX6MD7Z4Q, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1fokaL7frEkFdVZRGmqaq3TmZVNYpt4agWCJZ8ABHfsYwkjAuv, address:104.36.82.3:4792): protocol: rpc: dial tcp 104.36.82.3:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087741538 (85h15m0.810304578s) [noise=true , address=104.36.82.47:4537 , piece_id=UYZOMZFYI62UIZJFIZNBA6KYPW65MBSIGZTQWPTSB5AYG5NTJPLQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087744293 (85h15m0.810307333s) [noise=true , address=104.36.82.42:4829 , piece_id=EOF4BV45UFKWRBQTQWRLBLUPP5G67NWAXNENKI7ZCPCNB2R66ACQ, seg_pos={0 0}, upload=81779]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: write tcp 188.166.225.170:43972->104.36.82.42:4829: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:50:51.087768029 (85h15m0.810331069s) [seg_pos={0 0}, upload=81779]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:50:51.329260964 (85h15m1.051824004s) [upload=81779]: Done uploading segments... Jul 28 00:50:51.329282675 (85h15m1.051845715s) [upload=81779]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.24396658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.X..........X.....X.X...........X..................] duration: 5.839770412s FAIL (2025-07-28 00:50:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.839808255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X.......X.XX....X....X.......X.....X...............] duration: 5.820750402s FAIL (2025-07-28 00:50:47 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.820779757s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......................X............................] duration: 1.84565807s FAIL (2025-07-28 00:50:24 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.845686525s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.X..........X.....X.X...........X.................] duration: 5.983021674s FAIL (2025-07-28 00:50:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.983049168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X.......X.XX....X....X.......X.....X..............] duration: 5.893372821s FAIL (2025-07-28 00:50:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.893405033s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X.......X.XX....X....X.......X.....X.............] duration: 5.881630863s FAIL (2025-07-28 00:49:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.881658005s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.242071577s FAIL (2025-07-28 00:49:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== loading piece... Jul 28 00:48:51.275021562 (85h13m0.997584593s) [noise=true , address=104.36.82.50:4027 , piece_id=PB36YXNUQHTCLZVH3X7HTS2NBRB4LJ5UY4TBE6VIDMUS2M3NMFNA, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.275457617 (85h13m0.998020657s) [noise=true , address=104.36.82.5:4847 , piece_id=LS2CPKSJ3OTPOBTWHKCAT6BHRA275YRLFEICJDU7RHHZ3JI2NY4A, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.275889523 (85h13m0.998452563s) [noise=true , address=79.127.220.97:4375 , piece_id=375JMDQVFPME7EOLUBD3HN435QSPDFH6FMCBSKVRB2UI5OC4I56Q, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.277349897 (85h13m0.999912937s) [noise=true , address=104.36.82.41:4764 , piece_id=IOISLNZCQBBT4L5PFG66Q6W4RKGGFMQ55C2GOLM55HR7LIO6QW3A, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.278765596 (85h13m1.001328636s) [noise=true , address=104.36.82.20:4470 , piece_id=5CXBC2UYZXPZOH3IP2M42EZPN4NE3RGXFDCB2FWPKAGTDA5RSAMA, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.279038779 (85h13m1.001601819s) [noise=true , address=79.127.201.215:4352 , piece_id=L76L53UAAWK2CTAKRXNZRWHSKVFZ6F4ZAEMIOWUO5XWMGXTKLMQQ, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.279338282 (85h13m1.001901312s) [noise=true , address=104.36.82.51:4069 , piece_id=SITV7IVTNJK3AUVADO67PVY66FZRO25RDEHQJUP2A7KQJABIYAZQ, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.279645289 (85h13m1.002208329s) [noise=true , address=104.36.82.11:4134 , piece_id=BAUSKQ25YDAFAYYP6QPU45JD4BWD5STZONYQJ454FBY57UOPXSPA, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:51.279881300 (85h13m1.00244434s) [noise=true , address=104.36.82.42:4812 , piece_id=7Q3UW6WFNSIHFDBGUXJTHSHBDQUFR5HHCYWP6CD4UT4ECVHIS6UQ, seg_pos={0 0}, upload=81766]: Uploading piece... Jul 28 00:48:52.685134381 (85h13m2.407697422s) [upload=81766]: Next returned nil segment Jul 28 00:48:52.685155181 (85h13m2.407718222s) [upload=81766]: Waiting for error group managing segments... Jul 28 00:48:53.040284716 (85h13m2.762847756s) [noise=true , address=79.127.220.97:4375 , piece_id=375JMDQVFPME7EOLUBD3HN435QSPDFH6FMCBSKVRB2UI5OC4I56Q, seg_pos={0 0}, upload=81766]: Done uploading piece. err: <nil> Jul 28 00:48:53.206723041 (85h13m2.929286081s) [noise=true , address=104.36.82.5:4847 , piece_id=LS2CPKSJ3OTPOBTWHKCAT6BHRA275YRLFEICJDU7RHHZ3JI2NY4A, seg_pos={0 0}, upload=81766]: Done uploading piece. err: <nil> Jul 28 00:49:00.337488064 (85h13m10.060051104s) [noise=true , address=79.127.201.215:4352 , piece_id=L76L53UAAWK2CTAKRXNZRWHSKVFZ6F4ZAEMIOWUO5XWMGXTKLMQQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: <nil> Jul 28 00:49:50.175636429 (85h13m59.898199469s) [noise=true , address=104.36.82.13:4194 , piece_id=WXCHIB6HGFBZA73PDZ4ZZAE2IM5PYOFR75X4MPPMV3QVVLF7Z57Q, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:104.36.82.13:4194): protocol: rpc: dial tcp 104.36.82.13:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176027898 (85h13m59.898590938s) [noise=true , address=104.36.82.21:4494 , piece_id=4FXJTE44INMIQLKJ6KJ7WT2ZYKSST6QBM7SZUTFD5QK2PEWDJSTQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1sRaMuQG18B5SfPpzxp5VWjrwe38SS38g4askmbURxU9tepJHi, address:104.36.82.21:4494): protocol: rpc: dial tcp 104.36.82.21:4494: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176069497 (85h13m59.898632537s) [noise=true , address=104.36.82.4:4823 , piece_id=LME7UGRE24RARE3QR53LISSKLYZ5RJOHHX7LDUMRSKQMG6AGNLVQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1JWhB5QxbZsTD863tEfkEN1CUbKBMQvMJ9ZaiMAwKvxxXyDkEq, address:104.36.82.4:4823): protocol: rpc: dial tcp 104.36.82.4:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176203493 (85h13m59.898766543s) [noise=true , address=104.36.82.41:4764 , piece_id=IOISLNZCQBBT4L5PFG66Q6W4RKGGFMQ55C2GOLM55HR7LIO6QW3A, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176486344 (85h13m59.899049394s) [noise=true , address=104.36.82.11:4134 , piece_id=BAUSKQ25YDAFAYYP6QPU45JD4BWD5STZONYQJ454FBY57UOPXSPA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: write tcp 188.166.225.170:43398->104.36.82.11:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176519699 (85h13m59.899082739s) [noise=true , address=104.36.82.55:4201 , piece_id=FAS6SCHFP7OLFEZUUL45SCJX677SG2CEQNZNRMDPD74DTOHGQQ6Q, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176628407 (85h13m59.899191447s) [noise=true , address=104.36.82.52:4093 , piece_id=5KJHD2CH2M3ZAJXLQRZPHVGGZAFTDY3GNIQAK3CIDV4TD2KC5B2Q, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: write tcp 188.166.225.170:58120->104.36.82.52:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176712778 (85h13m59.899275818s) [noise=true , address=104.36.82.50:4027 , piece_id=PB36YXNUQHTCLZVH3X7HTS2NBRB4LJ5UY4TBE6VIDMUS2M3NMFNA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: rpc: dial tcp 104.36.82.50:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176780688 (85h13m59.899343728s) [noise=true , address=104.36.82.51:4069 , piece_id=SITV7IVTNJK3AUVADO67PVY66FZRO25RDEHQJUP2A7KQJABIYAZQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe, address:104.36.82.51:4069): protocol: write tcp 188.166.225.170:58972->104.36.82.51:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176818711 (85h13m59.899381751s) [noise=true , address=104.36.82.42:4812 , piece_id=7Q3UW6WFNSIHFDBGUXJTHSHBDQUFR5HHCYWP6CD4UT4ECVHIS6UQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1TmXeVTeejZvrWi2RPxMutM6kXuZiMCtPxQBwL4tq6mhoLwFQo, address:104.36.82.42:4812): protocol: write tcp 188.166.225.170:52326->104.36.82.42:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176848317 (85h13m59.899411358s) [noise=true , address=104.36.82.44:4896 , piece_id=MJ6NC2B52DJQULVKLOECW7A5G3CJ32CDQ7RIDCIU4I62POULOAYA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 188.166.225.170:37744->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176887001 (85h13m59.899450041s) [noise=true , address=104.36.82.54:4189 , piece_id=TQHXBE4YM6N5WGUPV56SHNSQNKM77AP3JSFNIXCZCLUGLEKHJPZA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: rpc: dial tcp 104.36.82.54:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176952496 (85h13m59.899515546s) [noise=true , address=104.36.82.53:4132 , piece_id=HJMWCQIBW7TGCWYPBHNEB6PICKLSYJAZCCOTWSDCX5VICDONS7YA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: write tcp 188.166.225.170:49304->104.36.82.53:4132: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.176988045 (85h13m59.899551085s) [noise=true , address=104.36.82.20:4470 , piece_id=5CXBC2UYZXPZOH3IP2M42EZPN4NE3RGXFDCB2FWPKAGTDA5RSAMA, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:12ZbWCEiPuqJqMhz88MJZrw7ar8wrV5abTTagADXTpnU8NQ8zYH, address:104.36.82.20:4470): protocol: rpc: dial tcp 104.36.82.20:4470: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.177036597 (85h13m59.899599637s) [noise=true , address=104.36.82.56:4253 , piece_id=EE7SQW377XIL2O34AMC3P22OBPJOUSWJOAB5PE7DKGULFCOCV3LQ, seg_pos={0 0}, upload=81766]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: write tcp 188.166.225.170:50860->104.36.82.56:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:49:50.177052147 (85h13m59.899615187s) [seg_pos={0 0}, upload=81766]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:49:50.415529002 (85h14m0.138092042s) [upload=81766]: Done uploading segments... Jul 28 00:49:50.415554239 (85h14m0.138117279s) [upload=81766]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.24208367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X.......................X...........................] duration: 2.183751521s FAIL (2025-07-28 00:49:40 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 2.183778875s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......................X..........................] duration: 1.84216544s FAIL (2025-07-28 00:49:18 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.842207652s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.X..........X.....X.X...........X................] duration: 5.908267016s FAIL (2025-07-28 00:49:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.90829576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [.................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234542888s FAIL (2025-07-28 00:48:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== P3DUOCOUBOFAPHJBRVA7AAZIRURKHDPPVZXUA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.824492279 (85h12m0.547055319s) [noise=true , address=104.36.82.41:4780 , piece_id=2WQEH7BT72ZLUICMUZKBXZW67WB6MV6PIC4IYUKAVIDZZZFD62GA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.824528007 (85h12m0.547091047s) [noise=true , address=79.127.201.215:4357 , piece_id=MTA4CHPP4ONRXJHYO4HENNFAJVDUBSLOV7CZEKDABU23F6OBQ32A, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.824729573 (85h12m0.547292613s) [noise=true , address=104.36.82.55:4196 , piece_id=XDII2ZSEQTM55BJ35S5J7XPBSHD4MGZJ2PCMMPFSBVR74W5U66BA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.824767195 (85h12m0.547330235s) [noise=true , address=104.36.82.53:4148 , piece_id=C7THPNW2BGNINLZVA2MICVENCSE7LLJHLVRSHAC5LNSUTLTN5NLA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.825034997 (85h12m0.547598037s) [noise=true , address=104.36.82.14:4231 , piece_id=MIXFIEDT6LBNSBJKKVLFIHD3O3OMKBUUPHRBKSPJW3VV4AVTRCLA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.825323740 (85h12m0.54788678s) [noise=true , address=104.36.82.16:4314 , piece_id=ZWLE4A6WS4FLDF35WAWZ3C4OFYSKRVHXW2J2S5HWT7BOC6W6UUWA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.825637760 (85h12m0.5482008s) [noise=true , address=104.36.82.42:4828 , piece_id=RVLHJ2QHKZRKRUODM7ATS66NC4Y6PKQ7DWRYM6F2YNAZSME2Q3DQ, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:51.937028269 (85h12m1.659591299s) [upload=81751]: Next returned nil segment Jul 28 00:47:51.937047556 (85h12m1.659610596s) [upload=81751]: Waiting for error group managing segments... Jul 28 00:47:52.309093908 (85h12m2.031656938s) [noise=true , address=79.127.205.234:4618 , piece_id=BREWR5ZI2WEGSYX7A4VK23PGFSGPWCOIQDP24I7KNUDSGUJI3TJQ, seg_pos={0 0}, upload=81751]: Done uploading piece. err: <nil> Jul 28 00:47:57.179542618 (85h12m6.902105658s) [noise=true , address=79.127.201.215:4357 , piece_id=MTA4CHPP4ONRXJHYO4HENNFAJVDUBSLOV7CZEKDABU23F6OBQ32A, seg_pos={0 0}, upload=81751]: Done uploading piece. err: <nil> Jul 28 00:48:49.729868761 (85h12m59.452431801s) [noise=true , address=104.36.82.15:4292 , piece_id=WVRCUVLAUYXXGZILE6WO3H45YSM2ZNN5MBJPK2V4DIFP4EAAOKRQ, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:104.36.82.15:4292): protocol: rpc: dial tcp 104.36.82.15:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.729923726 (85h12m59.452486766s) [noise=true , address=104.36.82.41:4780 , piece_id=2WQEH7BT72ZLUICMUZKBXZW67WB6MV6PIC4IYUKAVIDZZZFD62GA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:104.36.82.41:4780): protocol: rpc: dial tcp 104.36.82.41:4780: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.729969383 (85h12m59.452532423s) [noise=true , address=104.36.82.14:4231 , piece_id=MIXFIEDT6LBNSBJKKVLFIHD3O3OMKBUUPHRBKSPJW3VV4AVTRCLA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1ixY5pokE6MtrVH9FpKc1tGZhEAaW5brbzBeeGmYUczK3z7tCQ, address:104.36.82.14:4231): protocol: rpc: dial tcp 104.36.82.14:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730004020 (85h12m59.45256706s) [noise=true , address=104.36.82.16:4314 , piece_id=ZWLE4A6WS4FLDF35WAWZ3C4OFYSKRVHXW2J2S5HWT7BOC6W6UUWA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12KjL9YES3swMrgFERTQczzsc62AcXVJXC3sFuHsSt5M4DHt5Xw, address:104.36.82.16:4314): protocol: rpc: dial tcp 104.36.82.16:4314: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730062552 (85h12m59.452625592s) [noise=true , address=104.36.82.4:4811 , piece_id=CHYYWFJCJZAU4FGINEMLDC6XLTSRLFLXIQFALIY3EQRFBAKRIZBA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730096386 (85h12m59.452659426s) [noise=true , address=104.36.82.51:4073 , piece_id=CMHJN7GO6PDDNKEHCXNCBWPZC75LBXMSFDC4CHNXCOFT22W3B6YA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK, address:104.36.82.51:4073): protocol: rpc: dial tcp 104.36.82.51:4073: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730136142 (85h12m59.452699182s) [noise=true , address=104.36.82.42:4828 , piece_id=RVLHJ2QHKZRKRUODM7ATS66NC4Y6PKQ7DWRYM6F2YNAZSME2Q3DQ, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1cX7FtKC8KMopnX9zvkT6fYdnutK84cFjLW6xMwpXuGsqpiN8k, address:104.36.82.42:4828): protocol: write tcp 188.166.225.170:60530->104.36.82.42:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730162873 (85h12m59.452725913s) [noise=true , address=104.36.82.55:4196 , piece_id=XDII2ZSEQTM55BJ35S5J7XPBSHD4MGZJ2PCMMPFSBVR74W5U66BA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: write tcp 188.166.225.170:38440->104.36.82.55:4196: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730183212 (85h12m59.452746252s) [noise=true , address=104.36.82.54:4157 , piece_id=XKMAT7RSKKGK5EFMEWOF7XCZGXKASV7ZTNVKBTEPQIJAVSSA7JEA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: write tcp 188.166.225.170:57592->104.36.82.54:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730219311 (85h12m59.452782351s) [noise=true , address=104.36.82.52:4085 , piece_id=4Z6OBA2HHBORS55P3DUOCOUBOFAPHJBRVA7AAZIRURKHDPPVZXUA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:45628->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730223189 (85h12m59.452786229s) [noise=true , address=104.36.82.10:4108 , piece_id=CW3NEQE25H7QFR3VZJTXZYMT6UDL4TZKPRJCN3HHMQ4QXFSOMTXA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:104.36.82.10:4108): protocol: write tcp 188.166.225.170:45116->104.36.82.10:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730255000 (85h12m59.45281804s) [noise=true , address=104.36.82.53:4148 , piece_id=C7THPNW2BGNINLZVA2MICVENCSE7LLJHLVRSHAC5LNSUTLTN5NLA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: write tcp 188.166.225.170:35014->104.36.82.53:4148: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730274556 (85h12m59.452837596s) [noise=true , address=104.36.82.3:4796 , piece_id=RHINQ3DLTDF4HFWX3AW2XKXPAKWW52YLTWSUO3Z4CXUPPJNB2UQQ, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: write tcp 188.166.225.170:56424->104.36.82.3:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730291439 (85h12m59.452854479s) [noise=true , address=104.36.82.12:4164 , piece_id=TD7PDB2M3RFF2PVKRJOPWEN5BWO2LPJMY2T7PMZ4YHQJ2ZP7STJA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: write tcp 188.166.225.170:47366->104.36.82.12:4164: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730318511 (85h12m59.452881551s) [noise=true , address=104.36.82.52:4088 , piece_id=HB2Y42AYUKGBVWO2LIISIQQJNSK5ICMH7FKWVYAFB4BPV6UTUJJA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:104.36.82.52:4088): protocol: write tcp 188.166.225.170:49576->104.36.82.52:4088: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730340954 (85h12m59.452903994s) [noise=true , address=104.36.82.6:4884 , piece_id=V4KZADRDOGCKHOOW3LOY3SXYNSAJ5RFU7YGQMPQGVSW45Q7XJOPA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1c4THqUrQMeqY94EGiZSNKQDDnvycEZbGMrYxrb4s4XuaRe8br, address:104.36.82.6:4884): protocol: write tcp 188.166.225.170:53340->104.36.82.6:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730368457 (85h12m59.452931497s) [seg_pos={0 0}, upload=81751]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:48:49.962194256 (85h12m59.684757296s) [upload=81751]: Done uploading segments... Jul 28 00:48:49.962221167 (85h12m59.684784207s) [upload=81751]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.234567635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X.......X.XX....X....X.......X.....X............] duration: 5.84133658s FAIL (2025-07-28 00:48:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.841364063s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X.X..........X.....X.X...........X...............] duration: 5.933898055s FAIL (2025-07-28 00:48:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.93392711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......................X.........................] duration: 1.869117996s FAIL (2025-07-28 00:48:24 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.869141481s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X.......X.XX....X....X.......X.....X...........] duration: 5.718897998s FAIL (2025-07-28 00:47:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.71892449s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [..................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230436514s FAIL (2025-07-28 00:47:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ise=true , address=104.36.82.11:4133 , piece_id=Y5CW4L2NQ6GNPQZVMO4HPTCN5XORNIL5MI7ARL2WTIC3BEN7QE2Q, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.308356486 (85h10m59.030919526s) [noise=true , address=104.36.82.54:4189 , piece_id=GAQOZLDW3FHCXSDW3VH3LNDDHY5QL3DGKVR7I2U7NJBI563IXKTQ, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.308683772 (85h10m59.031246812s) [noise=true , address=104.36.82.15:4281 , piece_id=GFGYW2UH6YNG5JQ2JDCUVEJCSDZ4FD6Z7Z54YJQHGIONKKYB5FEA, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.308977224 (85h10m59.031540264s) [noise=true , address=104.36.82.14:4228 , piece_id=J2KFJJHU3GNOKO3MRDUM2XN7BVSIHXW7YNPD5TQOMT52ZSCYNNJQ, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.309605155 (85h10m59.032168195s) [noise=true , address=104.36.82.10:4096 , piece_id=H7JKLD5TCNLLNAXQ4A4CPBTGAJK3Z5VPM3KNEXZV5IYXZLIUSURQ, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.310038013 (85h10m59.032601053s) [noise=true , address=104.36.82.9:4033 , piece_id=V6SY33FJO4THYNVFD3HM7JIDLE5WTWKNXWXRHVTKMZF2OYTJ5ITA, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.310714738 (85h10m59.033277778s) [noise=true , address=104.36.82.13:4203 , piece_id=DBWSH7AIGUARAP5AIGOPMGLAXO2EOM7ZABT7QE73TYPTXDMM7MDQ, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.311061241 (85h10m59.033624281s) [noise=true , address=104.36.82.41:4778 , piece_id=AQ77RCBAKM3YMAF4O4DHH5WWKE2DZDU3WCF5F36IX6NH52BJUF6A, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.311654055 (85h10m59.034217095s) [noise=true , address=104.36.82.40:4965 , piece_id=MWA6HELA4R7DEWRXVVYYX2HKOXKHIJIUGEFNAR5FYLMCFELVCH2A, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:50.816271418 (85h11m0.538834459s) [upload=81736]: Next returned nil segment Jul 28 00:46:50.816291046 (85h11m0.538854086s) [upload=81736]: Waiting for error group managing segments... Jul 28 00:46:54.564772504 (85h11m4.287335584s) [noise=true , address=64.49.13.138:4483 , piece_id=OPQIO74C5MWPYGABCR2KIEAGHGR35AYRYCAW4DI7FFY2XKQC4XRA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: <nil> Jul 28 00:46:59.804875411 (85h11m9.527438461s) [noise=true , address=79.127.201.215:4349 , piece_id=U4JOTVOB2IUI2CCNNBQR7JDBLTLB4QVKS6XIJNBILGT2T7EB4POA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: <nil> Jul 28 00:47:48.218395891 (85h11m57.940958931s) [noise=true , address=104.36.82.52:4102 , piece_id=GG7FMXCSVVSIII2FKBYCF2WBW6D4VI2EZ3WUFMNCZL52TRFOVNBQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: rpc: dial tcp 104.36.82.52:4102: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218455796 (85h11m57.941018836s) [noise=true , address=104.36.82.42:4803 , piece_id=2GOY3TITYM2OI7NLDTU35LRNIVV5IOC2ATXXSSQYPXWF664CTO2Q, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:104.36.82.42:4803): protocol: rpc: dial tcp 104.36.82.42:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218488187 (85h11m57.941051217s) [noise=true , address=104.36.82.11:4133 , piece_id=Y5CW4L2NQ6GNPQZVMO4HPTCN5XORNIL5MI7ARL2WTIC3BEN7QE2Q, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: rpc: dial tcp 104.36.82.11:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218509789 (85h11m57.941072829s) [noise=true , address=104.36.82.9:4033 , piece_id=V6SY33FJO4THYNVFD3HM7JIDLE5WTWKNXWXRHVTKMZF2OYTJ5ITA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:4033): protocol: rpc: dial tcp 104.36.82.9:4033: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218530509 (85h11m57.941093539s) [noise=true , address=104.36.82.15:4281 , piece_id=GFGYW2UH6YNG5JQ2JDCUVEJCSDZ4FD6Z7Z54YJQHGIONKKYB5FEA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2, address:104.36.82.15:4281): protocol: rpc: dial tcp 104.36.82.15:4281: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218562790 (85h11m57.941125821s) [noise=true , address=104.36.82.41:4778 , piece_id=AQ77RCBAKM3YMAF4O4DHH5WWKE2DZDU3WCF5F36IX6NH52BJUF6A, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV, address:104.36.82.41:4778): protocol: rpc: dial tcp 104.36.82.41:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218587548 (85h11m57.941150588s) [noise=true , address=104.36.82.13:4203 , piece_id=DBWSH7AIGUARAP5AIGOPMGLAXO2EOM7ZABT7QE73TYPTXDMM7MDQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:18cNSsgqn3n6z5LxRcVCRzDzfFDpvptV1MKh6NdNRLW4bg9fWM, address:104.36.82.13:4203): protocol: rpc: dial tcp 104.36.82.13:4203: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218611113 (85h11m57.941174153s) [noise=true , address=104.36.82.44:4869 , piece_id=4WCILWRWHGQA4OWYF4HDKX6U6NQ7ETTVBH52MWQQCZR4DGUY6TKQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:104.36.82.44:4869): protocol: rpc: dial tcp 104.36.82.44:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218635310 (85h11m57.94119835s) [noise=true , address=104.36.82.54:4189 , piece_id=GAQOZLDW3FHCXSDW3VH3LNDDHY5QL3DGKVR7I2U7NJBI563IXKTQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: rpc: dial tcp 104.36.82.54:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218655768 (85h11m57.941218808s) [noise=true , address=104.36.82.6:4899 , piece_id=UJ3QCGBRRVD7QHXC6BICPXHTH6XCKM4BAXQ4CJKCWRJ5NG2S6K2Q, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: rpc: dial tcp 104.36.82.6:4899: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218678031 (85h11m57.941241071s) [noise=true , address=104.36.82.14:4228 , piece_id=J2KFJJHU3GNOKO3MRDUM2XN7BVSIHXW7YNPD5TQOMT52ZSCYNNJQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: rpc: dial tcp 104.36.82.14:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218699362 (85h11m57.941262402s) [noise=true , address=104.36.82.55:4203 , piece_id=BZ3PQU4AZA5CILSNRVVTEGUF2MO2YV275JGWN4KSHYBDYIF3FIFQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU, address:104.36.82.55:4203): protocol: rpc: dial tcp 104.36.82.55:4203: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218712697 (85h11m57.941275737s) [noise=true , address=104.36.82.10:4096 , piece_id=H7JKLD5TCNLLNAXQ4A4CPBTGAJK3Z5VPM3KNEXZV5IYXZLIUSURQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: rpc: dial tcp 104.36.82.10:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218742955 (85h11m57.941305985s) [noise=true , address=104.36.82.5:4866 , piece_id=VKKZW44M7CKZJEKJBSEPMH373H4TLWWXVUCYJYFXHXWBQJO3IILA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 188.166.225.170:48178->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218756922 (85h11m57.941319962s) [noise=true , address=104.36.82.40:4965 , piece_id=MWA6HELA4R7DEWRXVVYYX2HKOXKHIJIUGEFNAR5FYLMCFELVCH2A, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1ToFTqjRAWvpRQzf6vyHKDpforPyb4LbDGcCVhRhkwiBGKihnD, address:104.36.82.40:4965): protocol: write tcp 188.166.225.170:46974->104.36.82.40:4965: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218766750 (85h11m57.94132979s) [noise=true , address=104.36.82.12:4159 , piece_id=RXACT2D2ORFZ4OCFYT73WH7YTJ7HKOK7WR3YQ6G7HNIS37XXDF5Q, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: write tcp 188.166.225.170:58666->104.36.82.12:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218789254 (85h11m57.941352294s) [seg_pos={0 0}, upload=81736]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:47:48.447223334 (85h11m58.169786374s) [upload=81736]: Done uploading segments... Jul 28 00:47:48.447248201 (85h11m58.169811241s) [upload=81736]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.23045018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X.X..........X.....X.X...........X..............] duration: 5.804084816s FAIL (2025-07-28 00:47:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.804111419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......................X........................] duration: 1.868194931s FAIL (2025-07-28 00:47:22 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.868228074s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X.......X.XX....X....X.......X.....X..........] duration: 5.741435618s FAIL (2025-07-28 00:47:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.741467992s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X.X..........X.....X.X...........X.............] duration: 5.875800167s FAIL (2025-07-28 00:46:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.875830334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [...................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232790014s FAIL (2025-07-28 00:46:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== [noise=true , address=104.36.82.42:4821 , piece_id=PGVY7G42RSSXOMLBYF65H6THBBAEPG74WR7JACAWNGAHAG3OQ7NQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.810365758 (85h9m58.532928798s) [noise=true , address=104.36.82.40:4971 , piece_id=DYFZ7K7DEX5OYIR7A2HWBTXDCPI6Y5BBXEWBC6S5OB3WHRRGXZJQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.810599745 (85h9m58.533162775s) [noise=true , address=104.36.82.49:4992 , piece_id=VOHZXDUQJWHPEVIET3LQUMYQIFWQUP75UDUA4AVCF7J4UO3VK5JA, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.810818273 (85h9m58.533381313s) [noise=true , address=104.36.82.14:4242 , piece_id=DPAD7IVXUNT3LRHWKNOOIA72QRRWBLE5FW7D4A2HOGZ6HUG6OKQA, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811031561 (85h9m58.533594601s) [noise=true , address=104.36.82.13:4194 , piece_id=O54LDQ5TUJ7G7NC7ZPXVFLS4RNHQX5SLRSL3FHNODD5UGOQA7WSQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811222246 (85h9m58.533785286s) [noise=true , address=104.36.82.54:4179 , piece_id=FPUMX7NO22OSSSXCYBFRNFKU67HP4IOG6ENHY45F3EGHAKPEXBBQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811432288 (85h9m58.533995318s) [noise=true , address=104.36.82.56:4235 , piece_id=VYXBYYD4UEP65672CE2HZ3LLJ6SKBXUYXDQGARIVTQFNU4Y4W2FQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811630728 (85h9m58.534193768s) [noise=true , address=104.36.82.41:4773 , piece_id=PTXD3EUPC7R2M4J3V47B6W4TXEV6VSB66HU5XFZKJDV5E2DFXURQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811837633 (85h9m58.534400673s) [noise=true , address=104.36.82.11:4145 , piece_id=DS7DX6FKGY4YUCHSJAYCPN5WTUPX3CBK4X3TUSOMDXWVTWRH7GEA, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:49.923978246 (85h9m59.646541287s) [upload=81725]: Next returned nil segment Jul 28 00:45:49.923997843 (85h9m59.646560924s) [upload=81725]: Waiting for error group managing segments... Jul 28 00:45:50.309514087 (85h10m0.032077157s) [noise=true , address=79.127.205.235:4688 , piece_id=PXHJ3DCV3PPOPV44TKZSWEBN5DHBVCQ3YADPOAD7YHJVNJ55SSEA, seg_pos={0 0}, upload=81725]: Done uploading piece. err: <nil> Jul 28 00:45:50.549932185 (85h10m0.272495235s) [noise=true , address=79.127.220.97:4375 , piece_id=Z7KYI2LTSJ3LGBTDOTS7WDUXB34VQXH5JCCFFJBA6FKQ2CGYIJFQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: <nil> Jul 28 00:46:47.718547677 (85h10m57.441110717s) [noise=true , address=104.36.82.50:4021 , piece_id=ZGODXLN7KFRHTZO7BUUR5PAPMX7ATAR7OFY5NFE4GAC446X2NZ5Q, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1nz7Gxp4YTWuB1JDnjE5Q9BfVUUvyZ4SMBbZrxVQHrSUvcJbAQ, address:104.36.82.50:4021): protocol: rpc: dial tcp 104.36.82.50:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718568196 (85h10m57.441131226s) [noise=true , address=104.36.82.12:4166 , piece_id=FX4A3FB4T7NZAYL2XJ4KES4MJI23VAGQGZ7CQYJIADRN5OMF7IKQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718600618 (85h10m57.441163658s) [noise=true , address=104.36.82.13:4194 , piece_id=O54LDQ5TUJ7G7NC7ZPXVFLS4RNHQX5SLRSL3FHNODD5UGOQA7WSQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:104.36.82.13:4194): protocol: rpc: dial tcp 104.36.82.13:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718619914 (85h10m57.441182954s) [noise=true , address=104.36.82.42:4821 , piece_id=PGVY7G42RSSXOMLBYF65H6THBBAEPG74WR7JACAWNGAHAG3OQ7NQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: rpc: dial tcp 104.36.82.42:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718639292 (85h10m57.441202332s) [noise=true , address=104.36.82.11:4145 , piece_id=DS7DX6FKGY4YUCHSJAYCPN5WTUPX3CBK4X3TUSOMDXWVTWRH7GEA, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: rpc: dial tcp 104.36.82.11:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718659260 (85h10m57.4412223s) [noise=true , address=104.36.82.49:4992 , piece_id=VOHZXDUQJWHPEVIET3LQUMYQIFWQUP75UDUA4AVCF7J4UO3VK5JA, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv, address:104.36.82.49:4992): protocol: write tcp 188.166.225.170:34630->104.36.82.49:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718677534 (85h10m57.441240574s) [noise=true , address=104.36.82.47:4535 , piece_id=TAWFMYURBQRXOUHICDDDFYKD46EJZVOD64QPLPVYYIG6WI6BRL6A, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718693505 (85h10m57.441256545s) [noise=true , address=104.36.82.14:4242 , piece_id=DPAD7IVXUNT3LRHWKNOOIA72QRRWBLE5FW7D4A2HOGZ6HUG6OKQA, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: rpc: dial tcp 104.36.82.14:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718712502 (85h10m57.441275542s) [noise=true , address=104.36.82.43:4851 , piece_id=54JSAT5N2PXHEAJDVMBCEN2CDJTTJSDW32QYIBN7QIHGII2ZZF7A, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1nvd5hX9Wuih1hAtNzK4XAHcaSoFARdfRJijxByE4TW5HSpy4b, address:104.36.82.43:4851): protocol: rpc: dial tcp 104.36.82.43:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718726338 (85h10m57.441289378s) [noise=true , address=104.36.82.54:4179 , piece_id=FPUMX7NO22OSSSXCYBFRNFKU67HP4IOG6ENHY45F3EGHAKPEXBBQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: rpc: dial tcp 104.36.82.54:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718747168 (85h10m57.441310208s) [noise=true , address=104.36.82.53:4153 , piece_id=3ATH3AA56U4366MCMZADEAZ63HGCDFVFQPIWLQI735EPKJD4A54Q, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: rpc: dial tcp 104.36.82.53:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718761696 (85h10m57.441324736s) [noise=true , address=104.36.82.56:4235 , piece_id=VYXBYYD4UEP65672CE2HZ3LLJ6SKBXUYXDQGARIVTQFNU4Y4W2FQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: rpc: dial tcp 104.36.82.56:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718780612 (85h10m57.441343652s) [noise=true , address=104.36.82.55:4201 , piece_id=Z6DY5KF6M2MRW24ORMVFPBWK63OKE2NULGZWNZA6STKRQ3KZ4VOQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718793828 (85h10m57.441356858s) [noise=true , address=104.36.82.21:4489 , piece_id=D6NJIWRDB7TDOYNUGBB3KNBQO4CFAVTRFDVQ74JEJBTNZNVFKL5A, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12fao48Sd4XTApMqhudk7rT4pk6FssZRoqNdoUnrkL2b8cyExc5, address:104.36.82.21:4489): protocol: rpc: dial tcp 104.36.82.21:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718827983 (85h10m57.441391023s) [noise=true , address=104.36.82.40:4971 , piece_id=DYFZ7K7DEX5OYIR7A2HWBTXDCPI6Y5BBXEWBC6S5OB3WHRRGXZJQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH, address:104.36.82.40:4971): protocol: write tcp 188.166.225.170:41456->104.36.82.40:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718855726 (85h10m57.441418766s) [noise=true , address=104.36.82.41:4773 , piece_id=PTXD3EUPC7R2M4J3V47B6W4TXEV6VSB66HU5XFZKJDV5E2DFXURQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV, address:104.36.82.41:4773): protocol: write tcp 188.166.225.170:56004->104.36.82.41:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718876305 (85h10m57.441439356s) [seg_pos={0 0}, upload=81725]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:46:47.949071585 (85h10m57.671634625s) [upload=81725]: Done uploading segments... Jul 28 00:46:47.949100551 (85h10m57.671663591s) [upload=81725]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.232805453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X.......X.XX....X....X.......X.....X.........] duration: 5.813182914s FAIL (2025-07-28 00:46:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.813211877s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X.X..........X.....X.X...........X............] duration: 5.848012612s FAIL (2025-07-28 00:46:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.848039112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X.......................X.......................] duration: 2.114996714s FAIL (2025-07-28 00:46:15 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 2.115030269s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [....................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.228107546s FAIL (2025-07-28 00:45:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== . Jul 28 00:44:48.305181443 (85h8m58.027744494s) [noise=true , address=104.36.82.13:4212 , piece_id=GVQVLQSBNNDX3FB6WI2THSGRNH64X5K5VQ4GY7NN3ETGUK3QEUDA, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305316652 (85h8m58.027879692s) [noise=true , address=104.36.82.12:4155 , piece_id=CHREES42XYFSM2VQ7ZI33BSRLFZDTNLLPQMD3RPBZG3MURQHLCNQ, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305476779 (85h8m58.028039819s) [noise=true , address=104.36.82.14:4250 , piece_id=PHVCH3PG62VUCPB7A5RMWF2WQCLG7M4DDSXPNXHNWP5LSBOIDXGQ, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305518338 (85h8m58.028081368s) [noise=true , address=104.36.82.5:4860 , piece_id=WTSQ3IMM7CQF6MAPOEXBBRMI6PASRV6CSO6P2RCKP5275AJHBCEA, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305713992 (85h8m58.028277032s) [noise=true , address=104.36.82.20:4448 , piece_id=QMSM7SN7FFOMBEGFIKDNO2Q2GADVZMRSUCIPQBFEDIOB3ND7EE4Q, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305727158 (85h8m58.028290198s) [noise=true , address=104.36.82.52:4089 , piece_id=4DSMBKFZRAVMZRB7SMPPFSDFFAAHLKQVCRTKFKQ47HRR4HXG7GNQ, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305958430 (85h8m58.02852147s) [noise=true , address=104.36.82.56:4248 , piece_id=GYM7GTJSISHYEQKMLOUMV6NQLCJH6DY2YJNCHVEAILNACIIG5EBQ, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305983728 (85h8m58.028546768s) [noise=true , address=104.36.82.47:4538 , piece_id=ZCEFGZEY36X4YO4PC2EQAPZKFKBSHMQAS3U5IIZ5UHSDFVS7XKZA, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:49.855716732 (85h8m59.578279772s) [upload=81711]: Next returned nil segment Jul 28 00:44:49.855731269 (85h8m59.578294319s) [upload=81711]: Waiting for error group managing segments... Jul 28 00:44:52.198343156 (85h9m1.920906196s) [noise=true , address=79.127.201.215:4349 , piece_id=YOG7A3B3JX5GYUZF4BW6DQ67BPQU76LCBIO5W6EVCORBKPYWAWMQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: <nil> Jul 28 00:45:47.218477590 (85h9m56.94104063s) [noise=true , address=104.36.82.42:4806 , piece_id=XITTX6NB2KEEBS7IQGUPJFUL2CHUUWXSAQQN37EAFMY4QF555UNA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1tWnxuTSo4yHAJUzvVFTmwPSNb8cBJJJ17BnKd5Z3HMYf66vSC, address:104.36.82.42:4806): protocol: rpc: dial tcp 104.36.82.42:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218534218 (85h9m56.941097258s) [noise=true , address=104.36.82.55:4199 , piece_id=CAKWJVRE6AGFHWZHHZ2RM37D7WC4GAFL7U4TJQAG5UUNCSOP4BUQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218609452 (85h9m56.941172492s) [noise=true , address=104.36.82.51:4068 , piece_id=CLDQKGGKXGZPYR6HU27W4SF44VSGPFZYQGVXQFOSTCOR3W5QPUDA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY, address:104.36.82.51:4068): protocol: write tcp 188.166.225.170:36946->104.36.82.51:4068: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218647915 (85h9m56.941210945s) [noise=true , address=104.36.82.14:4250 , piece_id=PHVCH3PG62VUCPB7A5RMWF2WQCLG7M4DDSXPNXHNWP5LSBOIDXGQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:104.36.82.14:4250): protocol: rpc: dial tcp 104.36.82.14:4250: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218682642 (85h9m56.941245682s) [noise=true , address=104.36.82.20:4448 , piece_id=QMSM7SN7FFOMBEGFIKDNO2Q2GADVZMRSUCIPQBFEDIOB3ND7EE4Q, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp, address:104.36.82.20:4448): protocol: rpc: dial tcp 104.36.82.20:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218709473 (85h9m56.941272513s) [noise=true , address=104.36.82.13:4212 , piece_id=GVQVLQSBNNDX3FB6WI2THSGRNH64X5K5VQ4GY7NN3ETGUK3QEUDA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218825335 (85h9m56.941388375s) [noise=true , address=104.36.82.5:4860 , piece_id=WTSQ3IMM7CQF6MAPOEXBBRMI6PASRV6CSO6P2RCKP5275AJHBCEA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: rpc: dial tcp 104.36.82.5:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218876113 (85h9m56.941439153s) [noise=true , address=104.36.82.56:4248 , piece_id=GYM7GTJSISHYEQKMLOUMV6NQLCJH6DY2YJNCHVEAILNACIIG5EBQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: rpc: dial tcp 104.36.82.56:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218915518 (85h9m56.941478558s) [noise=true , address=104.36.82.10:4087 , piece_id=MMTYSPZIK6X3RD3K5AWBYGDX4PMS52B3YJID5XWHR2SP4ELQJBFQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:4087): protocol: rpc: dial tcp 104.36.82.10:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218952889 (85h9m56.941515929s) [noise=true , address=104.36.82.7:4944 , piece_id=2IE32AGQZCWWVJEB6ZT2COEBWEIVANKMGR2VXWZTH3KLNNMRDA7A, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: rpc: dial tcp 104.36.82.7:4944: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218991634 (85h9m56.941554674s) [noise=true , address=104.36.82.38:4715 , piece_id=GL3QDV5KGRYKG4R3AUFKQHF5XB275SOPWRWGQ2XZFPJSVECNBW4A, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: rpc: dial tcp 104.36.82.38:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218998336 (85h9m56.941561376s) [noise=true , address=104.36.82.15:4276 , piece_id=U4X5E3TZCNQCYNIQMXZVDIW7ZXOHP26FDPY263CKR5PM6FC7RCJQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219029656 (85h9m56.941592696s) [noise=true , address=104.36.82.41:4785 , piece_id=QPJATXNQX3JWRVGPOJA66AT6PWDEQIWRVU2GRIBLTLTQJEMMEVHQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB, address:104.36.82.41:4785): protocol: rpc: dial tcp 104.36.82.41:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219045166 (85h9m56.941608206s) [noise=true , address=104.36.82.12:4155 , piece_id=CHREES42XYFSM2VQ7ZI33BSRLFZDTNLLPQMD3RPBZG3MURQHLCNQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:104.36.82.12:4155): protocol: rpc: dial tcp 104.36.82.12:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219065614 (85h9m56.941628654s) [noise=true , address=104.36.82.2:4733 , piece_id=FYJ3EGYVRLDT4D5LZBDFSDTAM2PTNMTZ5DP7MRNGYEJWXRLTS3TA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:104.36.82.2:4733): protocol: rpc: dial tcp 104.36.82.2:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219083850 (85h9m56.94164689s) [noise=true , address=104.36.82.52:4089 , piece_id=4DSMBKFZRAVMZRB7SMPPFSDFFAAHLKQVCRTKFKQ47HRR4HXG7GNQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: rpc: dial tcp 104.36.82.52:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219114628 (85h9m56.941677668s) [noise=true , address=104.36.82.47:4538 , piece_id=ZCEFGZEY36X4YO4PC2EQAPZKFKBSHMQAS3U5IIZ5UHSDFVS7XKZA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: write tcp 188.166.225.170:58758->104.36.82.47:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219153083 (85h9m56.941716123s) [seg_pos={0 0}, upload=81711]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:45:47.445254399 (85h9m57.167817439s) [upload=81711]: Done uploading segments... Jul 28 00:45:47.445282573 (85h9m57.167845603s) [upload=81711]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.228119239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X.X..........X.....X.X...........X...........] duration: 5.80951247s FAIL (2025-07-28 00:45:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.809540061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X.......X.XX....X....X.......X.....X........] duration: 5.749221185s FAIL (2025-07-28 00:45:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.749243368s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X............X.......................X......................] duration: 1.802221244s FAIL (2025-07-28 00:45:26 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.802248136s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X.......X.XX....X....X.......X.....X.......] duration: 5.784039362s FAIL (2025-07-28 00:44:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.784061764s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [.....................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236142373s FAIL (2025-07-28 00:44:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== d=IV3CSZAQDQTO62GPE66QER5I3PBHNQOFDDLKTLR2NNGTGJOB6B2Q, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.824773080 (85h7m57.54733611s) [noise=true , address=79.127.201.215:4349 , piece_id=ELFZHCVAISQ322A2JAZFT44BG24NSZMI7GYMKSBVC6SHEQ7XCDHA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825005044 (85h7m57.547568084s) [noise=true , address=104.36.82.10:4080 , piece_id=BCT46Q4SKOUOT6KOHP6CS7CWPTEZLYQUGG7CNP64HP3JC7OBGBAQ, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825188916 (85h7m57.547751976s) [noise=true , address=104.36.82.46:4509 , piece_id=3R6DJAYUAYEG37MEKQ4AHMPGL3E6J3NETCPPJXZIY6UZ46CSEUSA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825431049 (85h7m57.547994089s) [noise=true , address=104.36.82.14:4226 , piece_id=FPNBHXD7FC6MBPBT6HRSOLUST4VAHL457QYW4H3IN6WXTQX3LMTA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825655098 (85h7m57.548218128s) [noise=true , address=104.36.82.56:4242 , piece_id=VOILM4KWS4P7ALU243GOMGE64WSZC6UVPJMFSTMEG6XOMG2ZEZRA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825669785 (85h7m57.548232815s) [noise=true , address=104.36.82.11:4133 , piece_id=55UVZDKUXL6D7FPVNN6WECBREEOTXE23UPQQBWQZIKENCURWKHHA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:48.919517680 (85h7m58.64208072s) [upload=81695]: Next returned nil segment Jul 28 00:43:48.919535524 (85h7m58.642098564s) [upload=81695]: Waiting for error group managing segments... Jul 28 00:43:49.901029062 (85h7m59.623592092s) [noise=true , address=79.127.205.245:4247 , piece_id=5R6RXFCNCZ7HVKKAWY7XX3QL7NWTLFCB2P5BE3U25VKRD2V4A7FA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: <nil> Jul 28 00:43:52.019794638 (85h8m1.742357688s) [noise=true , address=79.127.201.215:4349 , piece_id=ELFZHCVAISQ322A2JAZFT44BG24NSZMI7GYMKSBVC6SHEQ7XCDHA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: <nil> Jul 28 00:44:46.723687223 (85h8m56.446250263s) [noise=true , address=104.36.82.41:4782 , piece_id=5BJF4XKOBT27XZEXXO235UCGI2HFJ5H7OYMW3JTBMYY3IQH4NVTQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1vst2WW2D3NGbxPDDkjFQPxuwX3joAY77EdWCGXxU3H1NVGgcr, address:104.36.82.41:4782): protocol: write tcp 188.166.225.170:46992->104.36.82.41:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723735055 (85h8m56.446298095s) [noise=true , address=104.36.82.20:4459 , piece_id=EVENFGHQIU4HMOGOXG6XN2YO2QKC2Q7ZDKPTZDHDIFDPGNRLXWYQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: rpc: dial tcp 104.36.82.20:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723755433 (85h8m56.446318473s) [noise=true , address=104.36.82.54:4165 , piece_id=HGG4JDZZ4V7BB6Y7FQNKJDFO2V3ZBLDXW6C3GRBJAEQZD47C6P7Q, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: write tcp 188.166.225.170:33922->104.36.82.54:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723829265 (85h8m56.446392305s) [noise=true , address=104.36.82.6:4883 , piece_id=2L5DBUVO645XSL3LB2LPFK72LLALLUBNUBGN242DA6FST7BYH42Q, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12o2mgPVCn34Zny7hdDtGfQQwxXWKaEfKRXGrtkuat6mMstZiZ1, address:104.36.82.6:4883): protocol: write tcp 188.166.225.170:54914->104.36.82.6:4883: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723863691 (85h8m56.446426721s) [noise=true , address=104.36.82.21:4491 , piece_id=IV3CSZAQDQTO62GPE66QER5I3PBHNQOFDDLKTLR2NNGTGJOB6B2Q, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12Qxcfg1HCcyaydtJb4i3CtKaweXzyRZxkght7xRePgMzXX1QF6, address:104.36.82.21:4491): protocol: write tcp 188.166.225.170:46800->104.36.82.21:4491: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723931852 (85h8m56.446494892s) [noise=true , address=104.36.82.10:4080 , piece_id=BCT46Q4SKOUOT6KOHP6CS7CWPTEZLYQUGG7CNP64HP3JC7OBGBAQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3, address:104.36.82.10:4080): protocol: write tcp 188.166.225.170:37804->104.36.82.10:4080: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723970174 (85h8m56.446533215s) [noise=true , address=104.36.82.46:4509 , piece_id=3R6DJAYUAYEG37MEKQ4AHMPGL3E6J3NETCPPJXZIY6UZ46CSEUSA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: write tcp 188.166.225.170:44888->104.36.82.46:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724039918 (85h8m56.446603019s) [noise=true , address=104.36.82.56:4242 , piece_id=VOILM4KWS4P7ALU243GOMGE64WSZC6UVPJMFSTMEG6XOMG2ZEZRA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: write tcp 188.166.225.170:53882->104.36.82.56:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724073713 (85h8m56.446636753s) [noise=true , address=104.36.82.55:4227 , piece_id=V3RLTAWSPAFHFWC7UUV7BNCBBQRCUXNM2LTMZUECG67H7ZWII5XQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:19qGGwWT9xUYsJyRnFHyHKPFr6g4AsckxRNVTg2shfVhoQtcCz, address:104.36.82.55:4227): protocol: write tcp 188.166.225.170:41706->104.36.82.55:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724104081 (85h8m56.446667111s) [noise=true , address=104.36.82.13:4215 , piece_id=HUKA5NZGDYCHGYQ2INUAHRB3GSHOH34KNQQ7LUFVK4OGSICLGO3A, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1vV1hGhZqeEoh5ei7yRu49oyAvTZ2T7S1x3ooufJPtFH2TGBkQ, address:104.36.82.13:4215): protocol: write tcp 188.166.225.170:50148->104.36.82.13:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724171099 (85h8m56.446734139s) [noise=true , address=104.36.82.12:4152 , piece_id=FYDETR5M3DV52EMJS5KI37W2KR7TO2Y7SWEIBKRETT2USGQAODOA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:104.36.82.12:4152): protocol: write tcp 188.166.225.170:41576->104.36.82.12:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724201607 (85h8m56.446764637s) [noise=true , address=104.36.82.14:4226 , piece_id=FPNBHXD7FC6MBPBT6HRSOLUST4VAHL457QYW4H3IN6WXTQX3LMTA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:104.36.82.14:4226): protocol: write tcp 188.166.225.170:47274->104.36.82.14:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724265179 (85h8m56.446828219s) [noise=true , address=104.36.82.11:4133 , piece_id=55UVZDKUXL6D7FPVNN6WECBREEOTXE23UPQQBWQZIKENCURWKHHA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: write tcp 188.166.225.170:43966->104.36.82.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724304764 (85h8m56.446867805s) [noise=true , address=104.36.82.53:4152 , piece_id=NNU6SV4MWUH62AA4WYJGHUPTTQ3HHJYTBNJ45O4YGOJAZRAXYVCQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 188.166.225.170:50952->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724341045 (85h8m56.446904075s) [noise=true , address=104.36.82.41:4781 , piece_id=CPZA7NVSTTNPBLUDLZPANKINOWBIBWRR2PQI7B3ZVLCHBUQNAMCQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1QhCkTzSogXFKLmFV4wQUcALhvm9YhCSQudLxSrvQWmepheiFi, address:104.36.82.41:4781): protocol: write tcp 188.166.225.170:42670->104.36.82.41:4781: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724353057 (85h8m56.446916097s) [noise=true , address=104.36.82.42:4806 , piece_id=UR4V3CQ5LKZHDUPST6GOXMPDMJV4R65KSMPQQ42MJ3Q744CEM7ZQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1tWnxuTSo4yHAJUzvVFTmwPSNb8cBJJJ17BnKd5Z3HMYf66vSC, address:104.36.82.42:4806): protocol: write tcp 188.166.225.170:38162->104.36.82.42:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724376472 (85h8m56.446939512s) [seg_pos={0 0}, upload=81695]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:44:46.958002645 (85h8m56.680565685s) [upload=81695]: Done uploading segments... Jul 28 00:44:46.958029457 (85h8m56.680592497s) [upload=81695]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.236158894s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)