Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X..........X.....X...X...X....] duration: 8.09907452s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:33:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 8.099109888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X..........X.....X...X...X...] duration: 9.012881047s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:32:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 9.012907748s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...........................XXXXX..............XXXXXXXXXX....] duration: 23.157562201s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:32:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.157589464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.......X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......] duration: 5.001326296s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:31:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00135452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X..........X.....X...X...X..] duration: 8.826861302s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:31:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 8.82688608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [........X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X.....] duration: 4.321096124s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:31:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.321149556s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [............................XXXXX..............XXXXXXXXXX...] duration: 8.641251372s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:31:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.641267353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X..........X.....X...X...X.] duration: 9.573266348s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:30:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 9.573290803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X..........X.....X...X...X] duration: 30.004129062s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:30:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 30.00415391s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.........X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X....] duration: 4.103372891s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.103400545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.............................XXXXX..............XXXXXXXXXX..] duration: 10.464943354s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:30:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.464968824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X..........X.....X...X...] duration: 7.826480049s trace: https://jaeger.dev.storj.io/trace/e030e7ccfe74169 FAIL (2026-04-17 22:29:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37959 (246h40m43.264673983s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:26:56.445442749 (246h40m43.264678783s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:26:56.680984391 (246h40m43.500220415s) [seg_pos={0 0}, upload=236828]: Begin upload segment... Apr 17 22:26:56.681048645 (246h40m43.50028469s) [seg_pos={0 0}, upload=236828]: Done begin upload segment. Apr 17 22:26:56.681083343 (246h40m43.500319387s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681097750 (246h40m43.500333784s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681203023 (246h40m43.500439047s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681979182 (246h40m43.501215226s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.682540013 (246h40m43.501776047s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.683355678 (246h40m43.502591702s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684087060 (246h40m43.503323084s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684591082 (246h40m43.503827106s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.686358003 (246h40m43.505594027s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.688665065 (246h40m43.507901089s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.690702617 (246h40m43.509938652s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.692496279 (246h40m43.511732303s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.695655366 (246h40m43.51489139s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.697242800 (246h40m43.516478834s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.699718578 (246h40m43.518954612s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.701391005 (246h40m43.52062703s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:58.503644796 (246h40m45.32288082s) [upload=236828]: Next returned nil segment Apr 17 22:26:58.503661007 (246h40m45.322897041s) [upload=236828]: Waiting for error group managing segments... Apr 17 22:26:58.806983955 (246h40m45.626219979s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:58.825413052 (246h40m45.644649076s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.698754768 (246h40m46.517990792s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.942061167 (246h40m46.761297211s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.989468271 (246h40m46.808704305s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.991320666 (246h40m46.81055669s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.371241425 (246h40m47.190477449s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.410211514 (246h40m47.229447538s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.443769760 (246h40m47.263005784s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.574700229 (246h40m47.393936253s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.616364549 (246h40m47.435600603s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.966164496 (246h40m47.78540051s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447637825 (246h40m48.266873859s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447658094 (246h40m48.266894118s) [seg_pos={0 0}, upload=236828]: Satisfied cohort requirements after 13 pieces Apr 17 22:27:01.447856296 (246h40m48.26709232s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection Apr 17 22:27:01.447865153 (246h40m48.267101187s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.447882577 (246h40m48.267118601s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection Apr 17 22:27:01.447890883 (246h40m48.267126907s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448081771 (246h40m48.267317805s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz): protocol: write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection; write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection Apr 17 22:27:01.448088484 (246h40m48.267324518s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448108674 (246h40m48.267344698s) [seg_pos={0 0}, upload=236828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:27:01.448191924 (246h40m48.267427958s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:27:01.448197464 (246h40m48.267433488s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:27:01.723713096 (246h40m48.542949141s) [upload=236828]: Done uploading segments... Apr 17 22:27:01.723737243 (246h40m48.542973277s) [upload=236828]: Upload finished. err: <nil> Duration: 7.826502211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..........X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X...] duration: 4.434090759s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:29:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.434166285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..............................XXXXX..............XXXXXXXXXX.] duration: 11.710016229s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:29:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.710081165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X..........X.....X...X..] duration: 9.151741779s trace: https://jaeger.dev.storj.io/trace/e030e7ccfe74169 FAIL (2026-04-17 22:28:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37959 (246h40m43.264673983s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:26:56.445442749 (246h40m43.264678783s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:26:56.680984391 (246h40m43.500220415s) [seg_pos={0 0}, upload=236828]: Begin upload segment... Apr 17 22:26:56.681048645 (246h40m43.50028469s) [seg_pos={0 0}, upload=236828]: Done begin upload segment. Apr 17 22:26:56.681083343 (246h40m43.500319387s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681097750 (246h40m43.500333784s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681203023 (246h40m43.500439047s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681979182 (246h40m43.501215226s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.682540013 (246h40m43.501776047s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.683355678 (246h40m43.502591702s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684087060 (246h40m43.503323084s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684591082 (246h40m43.503827106s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.686358003 (246h40m43.505594027s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.688665065 (246h40m43.507901089s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.690702617 (246h40m43.509938652s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.692496279 (246h40m43.511732303s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.695655366 (246h40m43.51489139s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.697242800 (246h40m43.516478834s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.699718578 (246h40m43.518954612s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.701391005 (246h40m43.52062703s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:58.503644796 (246h40m45.32288082s) [upload=236828]: Next returned nil segment Apr 17 22:26:58.503661007 (246h40m45.322897041s) [upload=236828]: Waiting for error group managing segments... Apr 17 22:26:58.806983955 (246h40m45.626219979s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:58.825413052 (246h40m45.644649076s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.698754768 (246h40m46.517990792s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.942061167 (246h40m46.761297211s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.989468271 (246h40m46.808704305s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.991320666 (246h40m46.81055669s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.371241425 (246h40m47.190477449s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.410211514 (246h40m47.229447538s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.443769760 (246h40m47.263005784s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.574700229 (246h40m47.393936253s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.616364549 (246h40m47.435600603s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.966164496 (246h40m47.78540051s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447637825 (246h40m48.266873859s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447658094 (246h40m48.266894118s) [seg_pos={0 0}, upload=236828]: Satisfied cohort requirements after 13 pieces Apr 17 22:27:01.447856296 (246h40m48.26709232s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection Apr 17 22:27:01.447865153 (246h40m48.267101187s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.447882577 (246h40m48.267118601s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection Apr 17 22:27:01.447890883 (246h40m48.267126907s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448081771 (246h40m48.267317805s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz): protocol: write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection; write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection Apr 17 22:27:01.448088484 (246h40m48.267324518s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448108674 (246h40m48.267344698s) [seg_pos={0 0}, upload=236828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:27:01.448191924 (246h40m48.267427958s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:27:01.448197464 (246h40m48.267433488s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:27:01.723713096 (246h40m48.542949141s) [upload=236828]: Done uploading segments... Apr 17 22:27:01.723737243 (246h40m48.542973277s) [upload=236828]: Upload finished. err: <nil> Duration: 9.151758222s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...........X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X..] duration: 4.620470495s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:28:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.620520199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...............................XXXXX..............XXXXXXXXXX] duration: 1m0.000997603s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001033192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X.] duration: 4.403269531s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:28:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.40329511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X..........X.....X...X.] duration: 7.65982004s trace: https://jaeger.dev.storj.io/trace/e030e7ccfe74169 FAIL (2026-04-17 22:28:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37959 (246h40m43.264673983s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:26:56.445442749 (246h40m43.264678783s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:26:56.680984391 (246h40m43.500220415s) [seg_pos={0 0}, upload=236828]: Begin upload segment... Apr 17 22:26:56.681048645 (246h40m43.50028469s) [seg_pos={0 0}, upload=236828]: Done begin upload segment. Apr 17 22:26:56.681083343 (246h40m43.500319387s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681097750 (246h40m43.500333784s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681203023 (246h40m43.500439047s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681979182 (246h40m43.501215226s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.682540013 (246h40m43.501776047s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.683355678 (246h40m43.502591702s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684087060 (246h40m43.503323084s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684591082 (246h40m43.503827106s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.686358003 (246h40m43.505594027s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.688665065 (246h40m43.507901089s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.690702617 (246h40m43.509938652s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.692496279 (246h40m43.511732303s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.695655366 (246h40m43.51489139s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.697242800 (246h40m43.516478834s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.699718578 (246h40m43.518954612s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.701391005 (246h40m43.52062703s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:58.503644796 (246h40m45.32288082s) [upload=236828]: Next returned nil segment Apr 17 22:26:58.503661007 (246h40m45.322897041s) [upload=236828]: Waiting for error group managing segments... Apr 17 22:26:58.806983955 (246h40m45.626219979s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:58.825413052 (246h40m45.644649076s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.698754768 (246h40m46.517990792s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.942061167 (246h40m46.761297211s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.989468271 (246h40m46.808704305s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.991320666 (246h40m46.81055669s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.371241425 (246h40m47.190477449s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.410211514 (246h40m47.229447538s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.443769760 (246h40m47.263005784s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.574700229 (246h40m47.393936253s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.616364549 (246h40m47.435600603s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.966164496 (246h40m47.78540051s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447637825 (246h40m48.266873859s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447658094 (246h40m48.266894118s) [seg_pos={0 0}, upload=236828]: Satisfied cohort requirements after 13 pieces Apr 17 22:27:01.447856296 (246h40m48.26709232s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection Apr 17 22:27:01.447865153 (246h40m48.267101187s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.447882577 (246h40m48.267118601s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection Apr 17 22:27:01.447890883 (246h40m48.267126907s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448081771 (246h40m48.267317805s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz): protocol: write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection; write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection Apr 17 22:27:01.448088484 (246h40m48.267324518s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448108674 (246h40m48.267344698s) [seg_pos={0 0}, upload=236828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:27:01.448191924 (246h40m48.267427958s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:27:01.448197464 (246h40m48.267433488s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:27:01.723713096 (246h40m48.542949141s) [upload=236828]: Done uploading segments... Apr 17 22:27:01.723737243 (246h40m48.542973277s) [upload=236828]: Upload finished. err: <nil> Duration: 7.659893341s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X] duration: 5.000990072s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:28:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001007857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [................................XXXXX..............XXXXXXXXX] duration: 1m0.001720016s trace: https://jaeger.dev.storj.io/trace/532f681695e7f03 FAIL (2026-04-17 22:27:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001738201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X..........X.....X...X] duration: 30.00147431s trace: https://jaeger.dev.storj.io/trace/e030e7ccfe74169 FAIL (2026-04-17 22:27:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37959 (246h40m43.264673983s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:26:56.445442749 (246h40m43.264678783s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:26:56.680984391 (246h40m43.500220415s) [seg_pos={0 0}, upload=236828]: Begin upload segment... Apr 17 22:26:56.681048645 (246h40m43.50028469s) [seg_pos={0 0}, upload=236828]: Done begin upload segment. Apr 17 22:26:56.681083343 (246h40m43.500319387s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681097750 (246h40m43.500333784s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681203023 (246h40m43.500439047s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681979182 (246h40m43.501215226s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.682540013 (246h40m43.501776047s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.683355678 (246h40m43.502591702s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684087060 (246h40m43.503323084s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684591082 (246h40m43.503827106s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.686358003 (246h40m43.505594027s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.688665065 (246h40m43.507901089s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.690702617 (246h40m43.509938652s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.692496279 (246h40m43.511732303s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.695655366 (246h40m43.51489139s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.697242800 (246h40m43.516478834s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.699718578 (246h40m43.518954612s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.701391005 (246h40m43.52062703s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:58.503644796 (246h40m45.32288082s) [upload=236828]: Next returned nil segment Apr 17 22:26:58.503661007 (246h40m45.322897041s) [upload=236828]: Waiting for error group managing segments... Apr 17 22:26:58.806983955 (246h40m45.626219979s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:58.825413052 (246h40m45.644649076s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.698754768 (246h40m46.517990792s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.942061167 (246h40m46.761297211s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.989468271 (246h40m46.808704305s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.991320666 (246h40m46.81055669s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.371241425 (246h40m47.190477449s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.410211514 (246h40m47.229447538s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.443769760 (246h40m47.263005784s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.574700229 (246h40m47.393936253s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.616364549 (246h40m47.435600603s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.966164496 (246h40m47.78540051s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447637825 (246h40m48.266873859s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447658094 (246h40m48.266894118s) [seg_pos={0 0}, upload=236828]: Satisfied cohort requirements after 13 pieces Apr 17 22:27:01.447856296 (246h40m48.26709232s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection Apr 17 22:27:01.447865153 (246h40m48.267101187s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.447882577 (246h40m48.267118601s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection Apr 17 22:27:01.447890883 (246h40m48.267126907s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448081771 (246h40m48.267317805s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz): protocol: write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection; write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection Apr 17 22:27:01.448088484 (246h40m48.267324518s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448108674 (246h40m48.267344698s) [seg_pos={0 0}, upload=236828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:27:01.448191924 (246h40m48.267427958s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:27:01.448197464 (246h40m48.267433488s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:27:01.723713096 (246h40m48.542949141s) [upload=236828]: Done uploading segments... Apr 17 22:27:01.723737243 (246h40m48.542973277s) [upload=236828]: Upload finished. err: <nil> Duration: 30.001506042s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.] duration: 4.318718346s trace: https://jaeger.dev.storj.io/trace/366a6dd433456773 FAIL (2026-04-17 22:27:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.318745969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X] duration: 5.001556725s trace: https://jaeger.dev.storj.io/trace/366a6dd433456773 FAIL (2026-04-17 22:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001584729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X..........X.....X...] duration: 18.900543393s trace: https://jaeger.dev.storj.io/trace/42b424317d5a4121 FAIL (2026-04-17 22:26:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:24:05.694992332 (246h37m52.514228356s) [upload=236781]: Starting upload Apr 17 22:24:05.695045855 (246h37m52.514281879s) [upload=236781]: Done starting upload Apr 17 22:24:05.695147542 (246h37m52.514383566s) [upload=236781]: Uploading segments... Apr 17 22:24:05.695218739 (246h37m52.514454773s) [upload=236781]: Got next segment. Inline: false Apr 17 22:24:05.695247164 (246h37m52.514483188s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:24:05.695251513 (246h37m52.514487537s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:24:05.932814858 (246h37m52.752050872s) [seg_pos={0 0}, upload=236781]: Begin upload segment... Apr 17 22:24:05.932862179 (246h37m52.752098203s) [seg_pos={0 0}, upload=236781]: Done begin upload segment. Apr 17 22:24:05.932878050 (246h37m52.752114074s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.932886456 (246h37m52.75212248s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933244698 (246h37m52.752480722s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933248785 (246h37m52.752484809s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933532453 (246h37m52.752768477s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933584163 (246h37m52.752820187s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933898440 (246h37m52.753134464s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934250218 (246h37m52.753486232s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934363717 (246h37m52.753599741s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934643347 (246h37m52.753879371s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934869704 (246h37m52.754105728s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935128173 (246h37m52.754364197s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935374137 (246h37m52.754610161s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935623228 (246h37m52.754859252s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935978473 (246h37m52.755214497s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.936271699 (246h37m52.755507723s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:07.749332052 (246h37m54.568568076s) [upload=236781]: Next returned nil segment Apr 17 22:24:07.749345899 (246h37m54.568581923s) [upload=236781]: Waiting for error group managing segments... Apr 17 22:24:08.055489009 (246h37m54.874725033s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.060851439 (246h37m54.880087463s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.087574814 (246h37m54.906810838s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.218312049 (246h37m56.037548073s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.516129322 (246h37m56.335365346s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.574962518 (246h37m56.394198532s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.644416080 (246h37m56.463652104s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.696489518 (246h37m56.515725532s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.985990091 (246h37m56.805226105s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.022897720 (246h37m56.842133744s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.034800812 (246h37m56.854036836s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.120044970 (246h37m56.939280994s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133421156 (246h37m56.95265721s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133446624 (246h37m56.952682729s) [seg_pos={0 0}, upload=236781]: Satisfied cohort requirements after 13 pieces Apr 17 22:24:10.133736224 (246h37m56.952972328s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133750301 (246h37m56.952986405s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133763746 (246h37m56.95299977s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133825065 (246h37m56.953061149s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133966618 (246h37m56.953202662s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133976136 (246h37m56.953212161s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133995313 (246h37m56.953231348s) [seg_pos={0 0}, upload=236781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:24:10.134105837 (246h37m56.953341861s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:24:10.134112840 (246h37m56.953348864s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:24:10.433095060 (246h37m57.252331084s) [upload=236781]: Done uploading segments... Apr 17 22:24:10.433115701 (246h37m57.252351735s) [upload=236781]: Upload finished. err: <nil> Duration: 18.900715194s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.................................XXXXX..............XXXXXXXX] duration: 1m0.000797277s trace: https://jaeger.dev.storj.io/trace/19fe5fe1f7c99e30 FAIL (2026-04-17 22:26:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000809669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X..........X.....X..] duration: 7.81162255s trace: https://jaeger.dev.storj.io/trace/42b424317d5a4121 FAIL (2026-04-17 22:25:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:24:05.694992332 (246h37m52.514228356s) [upload=236781]: Starting upload Apr 17 22:24:05.695045855 (246h37m52.514281879s) [upload=236781]: Done starting upload Apr 17 22:24:05.695147542 (246h37m52.514383566s) [upload=236781]: Uploading segments... Apr 17 22:24:05.695218739 (246h37m52.514454773s) [upload=236781]: Got next segment. Inline: false Apr 17 22:24:05.695247164 (246h37m52.514483188s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:24:05.695251513 (246h37m52.514487537s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:24:05.932814858 (246h37m52.752050872s) [seg_pos={0 0}, upload=236781]: Begin upload segment... Apr 17 22:24:05.932862179 (246h37m52.752098203s) [seg_pos={0 0}, upload=236781]: Done begin upload segment. Apr 17 22:24:05.932878050 (246h37m52.752114074s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.932886456 (246h37m52.75212248s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933244698 (246h37m52.752480722s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933248785 (246h37m52.752484809s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933532453 (246h37m52.752768477s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933584163 (246h37m52.752820187s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933898440 (246h37m52.753134464s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934250218 (246h37m52.753486232s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934363717 (246h37m52.753599741s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934643347 (246h37m52.753879371s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934869704 (246h37m52.754105728s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935128173 (246h37m52.754364197s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935374137 (246h37m52.754610161s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935623228 (246h37m52.754859252s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935978473 (246h37m52.755214497s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.936271699 (246h37m52.755507723s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:07.749332052 (246h37m54.568568076s) [upload=236781]: Next returned nil segment Apr 17 22:24:07.749345899 (246h37m54.568581923s) [upload=236781]: Waiting for error group managing segments... Apr 17 22:24:08.055489009 (246h37m54.874725033s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.060851439 (246h37m54.880087463s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.087574814 (246h37m54.906810838s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.218312049 (246h37m56.037548073s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.516129322 (246h37m56.335365346s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.574962518 (246h37m56.394198532s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.644416080 (246h37m56.463652104s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.696489518 (246h37m56.515725532s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.985990091 (246h37m56.805226105s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.022897720 (246h37m56.842133744s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.034800812 (246h37m56.854036836s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.120044970 (246h37m56.939280994s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133421156 (246h37m56.95265721s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133446624 (246h37m56.952682729s) [seg_pos={0 0}, upload=236781]: Satisfied cohort requirements after 13 pieces Apr 17 22:24:10.133736224 (246h37m56.952972328s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133750301 (246h37m56.952986405s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133763746 (246h37m56.95299977s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133825065 (246h37m56.953061149s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133966618 (246h37m56.953202662s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133976136 (246h37m56.953212161s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133995313 (246h37m56.953231348s) [seg_pos={0 0}, upload=236781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:24:10.134105837 (246h37m56.953341861s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:24:10.134112840 (246h37m56.953348864s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:24:10.433095060 (246h37m57.252331084s) [upload=236781]: Done uploading segments... Apr 17 22:24:10.433115701 (246h37m57.252351735s) [upload=236781]: Upload finished. err: <nil> Duration: 7.811662106s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...] duration: 4.621319847s trace: https://jaeger.dev.storj.io/trace/579efa8346903a39 FAIL (2026-04-17 22:25:42 UTC) orbiter warning: context deadline exceeded Duration: 4.621344313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..................................XXXXX..............XXXXXXX] duration: 1m0.00082905s trace: https://jaeger.dev.storj.io/trace/30f9dc36da647e3c FAIL (2026-04-17 22:25:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00084001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X..........X.....X.] duration: 9.351278529s trace: https://jaeger.dev.storj.io/trace/42b424317d5a4121 FAIL (2026-04-17 22:25:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:24:05.694992332 (246h37m52.514228356s) [upload=236781]: Starting upload Apr 17 22:24:05.695045855 (246h37m52.514281879s) [upload=236781]: Done starting upload Apr 17 22:24:05.695147542 (246h37m52.514383566s) [upload=236781]: Uploading segments... Apr 17 22:24:05.695218739 (246h37m52.514454773s) [upload=236781]: Got next segment. Inline: false Apr 17 22:24:05.695247164 (246h37m52.514483188s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:24:05.695251513 (246h37m52.514487537s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:24:05.932814858 (246h37m52.752050872s) [seg_pos={0 0}, upload=236781]: Begin upload segment... Apr 17 22:24:05.932862179 (246h37m52.752098203s) [seg_pos={0 0}, upload=236781]: Done begin upload segment. Apr 17 22:24:05.932878050 (246h37m52.752114074s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.932886456 (246h37m52.75212248s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933244698 (246h37m52.752480722s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933248785 (246h37m52.752484809s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933532453 (246h37m52.752768477s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933584163 (246h37m52.752820187s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933898440 (246h37m52.753134464s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934250218 (246h37m52.753486232s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934363717 (246h37m52.753599741s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934643347 (246h37m52.753879371s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934869704 (246h37m52.754105728s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935128173 (246h37m52.754364197s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935374137 (246h37m52.754610161s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935623228 (246h37m52.754859252s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935978473 (246h37m52.755214497s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.936271699 (246h37m52.755507723s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:07.749332052 (246h37m54.568568076s) [upload=236781]: Next returned nil segment Apr 17 22:24:07.749345899 (246h37m54.568581923s) [upload=236781]: Waiting for error group managing segments... Apr 17 22:24:08.055489009 (246h37m54.874725033s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.060851439 (246h37m54.880087463s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.087574814 (246h37m54.906810838s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.218312049 (246h37m56.037548073s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.516129322 (246h37m56.335365346s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.574962518 (246h37m56.394198532s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.644416080 (246h37m56.463652104s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.696489518 (246h37m56.515725532s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.985990091 (246h37m56.805226105s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.022897720 (246h37m56.842133744s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.034800812 (246h37m56.854036836s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.120044970 (246h37m56.939280994s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133421156 (246h37m56.95265721s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133446624 (246h37m56.952682729s) [seg_pos={0 0}, upload=236781]: Satisfied cohort requirements after 13 pieces Apr 17 22:24:10.133736224 (246h37m56.952972328s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133750301 (246h37m56.952986405s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133763746 (246h37m56.95299977s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133825065 (246h37m56.953061149s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133966618 (246h37m56.953202662s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133976136 (246h37m56.953212161s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133995313 (246h37m56.953231348s) [seg_pos={0 0}, upload=236781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:24:10.134105837 (246h37m56.953341861s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:24:10.134112840 (246h37m56.953348864s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:24:10.433095060 (246h37m57.252331084s) [upload=236781]: Done uploading segments... Apr 17 22:24:10.433115701 (246h37m57.252351735s) [upload=236781]: Upload finished. err: <nil> Duration: 9.351302906s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [....X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X..] duration: 4.482347838s trace: https://jaeger.dev.storj.io/trace/579efa8346903a39 FAIL (2026-04-17 22:24:43 UTC) orbiter warning: context deadline exceeded Duration: 4.482381623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X..........X.....X] duration: 30.002993184s trace: https://jaeger.dev.storj.io/trace/42b424317d5a4121 FAIL (2026-04-17 22:24:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:24:05.694992332 (246h37m52.514228356s) [upload=236781]: Starting upload Apr 17 22:24:05.695045855 (246h37m52.514281879s) [upload=236781]: Done starting upload Apr 17 22:24:05.695147542 (246h37m52.514383566s) [upload=236781]: Uploading segments... Apr 17 22:24:05.695218739 (246h37m52.514454773s) [upload=236781]: Got next segment. Inline: false Apr 17 22:24:05.695247164 (246h37m52.514483188s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:24:05.695251513 (246h37m52.514487537s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:24:05.932814858 (246h37m52.752050872s) [seg_pos={0 0}, upload=236781]: Begin upload segment... Apr 17 22:24:05.932862179 (246h37m52.752098203s) [seg_pos={0 0}, upload=236781]: Done begin upload segment. Apr 17 22:24:05.932878050 (246h37m52.752114074s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.932886456 (246h37m52.75212248s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933244698 (246h37m52.752480722s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933248785 (246h37m52.752484809s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933532453 (246h37m52.752768477s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933584163 (246h37m52.752820187s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933898440 (246h37m52.753134464s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934250218 (246h37m52.753486232s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934363717 (246h37m52.753599741s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934643347 (246h37m52.753879371s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934869704 (246h37m52.754105728s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935128173 (246h37m52.754364197s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935374137 (246h37m52.754610161s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935623228 (246h37m52.754859252s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935978473 (246h37m52.755214497s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.936271699 (246h37m52.755507723s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:07.749332052 (246h37m54.568568076s) [upload=236781]: Next returned nil segment Apr 17 22:24:07.749345899 (246h37m54.568581923s) [upload=236781]: Waiting for error group managing segments... Apr 17 22:24:08.055489009 (246h37m54.874725033s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.060851439 (246h37m54.880087463s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.087574814 (246h37m54.906810838s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.218312049 (246h37m56.037548073s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.516129322 (246h37m56.335365346s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.574962518 (246h37m56.394198532s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.644416080 (246h37m56.463652104s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.696489518 (246h37m56.515725532s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.985990091 (246h37m56.805226105s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.022897720 (246h37m56.842133744s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.034800812 (246h37m56.854036836s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.120044970 (246h37m56.939280994s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133421156 (246h37m56.95265721s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133446624 (246h37m56.952682729s) [seg_pos={0 0}, upload=236781]: Satisfied cohort requirements after 13 pieces Apr 17 22:24:10.133736224 (246h37m56.952972328s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133750301 (246h37m56.952986405s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133763746 (246h37m56.95299977s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133825065 (246h37m56.953061149s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133966618 (246h37m56.953202662s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133976136 (246h37m56.953212161s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133995313 (246h37m56.953231348s) [seg_pos={0 0}, upload=236781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:24:10.134105837 (246h37m56.953341861s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:24:10.134112840 (246h37m56.953348864s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:24:10.433095060 (246h37m57.252331084s) [upload=236781]: Done uploading segments... Apr 17 22:24:10.433115701 (246h37m57.252351735s) [upload=236781]: Upload finished. err: <nil> Duration: 30.003013104s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...................................XXXXX..............XXXXXX] duration: 1m0.000690709s trace: https://jaeger.dev.storj.io/trace/5a1d9dd2adbb1be4 FAIL (2026-04-17 22:24:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000701299s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.....X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X.] duration: 4.831154779s trace: https://jaeger.dev.storj.io/trace/579efa8346903a39 FAIL (2026-04-17 22:23:34 UTC) orbiter warning: context deadline exceeded Duration: 4.831220396s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [....................................XXXXX..............XXXXX] duration: 1m0.001399423s trace: https://jaeger.dev.storj.io/trace/5bcabdc62a3d2b39 FAIL (2026-04-17 22:23:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001420324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [......X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X] duration: 5.000548138s trace: https://jaeger.dev.storj.io/trace/579efa8346903a39 FAIL (2026-04-17 22:23:19 UTC) orbiter warning: context deadline exceeded Duration: 5.000571112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..........X.....] duration: 9.395222237s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:23:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 9.395260079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..........X....] duration: 9.035738666s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:22:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 9.03575621s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.......X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.] duration: 4.169163326s trace: https://jaeger.dev.storj.io/trace/7c81e63b0bcc6d75 FAIL (2026-04-17 22:22:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.169189174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....................................XXXXX..............XXXX] duration: 1m0.002265994s trace: https://jaeger.dev.storj.io/trace/6a7e45e61a3c62fc FAIL (2026-04-17 22:22:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002281354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X..........X...] duration: 9.501303572s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:22:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 9.501327838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [........X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X] duration: 5.001093964s trace: https://jaeger.dev.storj.io/trace/7c81e63b0bcc6d75 FAIL (2026-04-17 22:21:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001116777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X..........X..] duration: 9.557344091s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:21:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 9.557374671s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.........X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..] duration: 5.000715004s trace: https://jaeger.dev.storj.io/trace/281ef638e16750f7 FAIL (2026-04-17 22:21:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000738239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......................................XXXXX..............XXX] duration: 1m0.00077685s trace: https://jaeger.dev.storj.io/trace/7e397d6b39319fe1 FAIL (2026-04-17 22:21:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000799695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X..........X.] duration: 8.41249953s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 8.412528136s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..........X............X.XX...X.......XXXX.X.XX.XX.XXXX...X.] duration: 4.73454751s trace: https://jaeger.dev.storj.io/trace/281ef638e16750f7 FAIL (2026-04-17 22:20:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.734603999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..........X] duration: 30.002871816s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:20:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 30.002892967s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.......................................XXXXX..............XX] duration: 1m0.00083783s trace: https://jaeger.dev.storj.io/trace/f85cf54b682d8b6 FAIL (2026-04-17 22:20:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00084788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...........X............X.XX...X.......XXXX.X.XX.XX.XXXX...X] duration: 5.001714438s trace: https://jaeger.dev.storj.io/trace/281ef638e16750f7 FAIL (2026-04-17 22:19:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001736572s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)