Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXX.........] duration: 5.670252797s FAIL (2025-07-29 18:16:31 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 5.670270232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.887339578s FAIL (2025-07-29 18:16:17 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.887370898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXX........] duration: 5.73847981s FAIL (2025-07-29 18:16:14 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 5.73850064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [...............X..........X.XX......XX.XX.XXXXX.X...........] duration: 5.57143589s FAIL (2025-07-29 18:16:14 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.571464614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.958912468s FAIL (2025-07-29 18:15:41 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.95895525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [................X..........X.XX......XX.XX.XXXXX.X..........] duration: 5.561897369s FAIL (2025-07-29 18:15:26 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.561927947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.883339425s FAIL (2025-07-29 18:15:18 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.883371486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXX.......] duration: 5.804023917s FAIL (2025-07-29 18:15:13 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 5.804055798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.................X..........X.XX......XX.XX.XXXXX.X.........] duration: 5.651329992s FAIL (2025-07-29 18:15:04 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.651361342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.905750022s FAIL (2025-07-29 18:14:33 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.905780289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXX......] duration: 5.759568028s FAIL (2025-07-29 18:14:18 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 5.759594188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..................X..........X.XX......XX.XX.XXXXX.X........] duration: 5.730247971s FAIL (2025-07-29 18:14:00 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.730270835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.988560838s FAIL (2025-07-29 18:13:50 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.988586066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [...................X..........X.XX......XX.XX.XXXXX.X.......] duration: 5.739995057s FAIL (2025-07-29 18:13:39 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.740023623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXX.....] duration: 5.836234014s FAIL (2025-07-29 18:13:33 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 5.836264391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................X..........X.XX......XX.XX.XXXXX.X......] duration: 5.919616455s FAIL (2025-07-29 18:12:43 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.919638267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.888882538s FAIL (2025-07-29 18:12:37 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.888911303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXX....] duration: 6.076591209s FAIL (2025-07-29 18:12:22 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 6.076615916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....................X..........X.XX......XX.XX.XXXXX.X.....] duration: 5.921379241s FAIL (2025-07-29 18:11:59 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.921408496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.220542315s FAIL (2025-07-29 18:11:49 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.220570269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [......................X..........X.XX......XX.XX.XXXXX.X....] duration: 6.004494313s FAIL (2025-07-29 18:11:33 UTC) orbiter warning: uplink: context deadline exceeded Duration: 6.004557574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXX...] duration: 5.776395407s FAIL (2025-07-29 18:11:30 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 5.776424223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......................X..........X.XX......XX.XX.XXXXX.X...] duration: 5.748889016s FAIL (2025-07-29 18:11:01 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.748920805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXX..] duration: 5.693694522s FAIL (2025-07-29 18:10:54 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 5.693721894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [........................X..........X.XX......XX.XX.XXXXX.X..] duration: 6.069732226s FAIL (2025-07-29 18:10:38 UTC) orbiter warning: uplink: context deadline exceeded Duration: 6.069780708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X.................................X.........................] duration: 2.046161058s FAIL (2025-07-29 18:10:37 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.046195604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X.................................X........................] duration: 1.894969317s FAIL (2025-07-29 18:10:15 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.895016547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXX.] duration: 22.527538198s FAIL (2025-07-29 18:10:08 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 22.527558496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........................X..........X.XX......XX.XX.XXXXX.X.] duration: 8.505100826s FAIL (2025-07-29 18:10:05 UTC) orbiter warning: uplink: context deadline exceeded Duration: 8.505126185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 30.002275971s FAIL (2025-07-29 18:09:42 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Jul 29 18:09:13.411258846 (25h23m25.887197311s) [upload=24302]: Starting upload Jul 29 18:09:13.411327648 (25h23m25.887266113s) [upload=24302]: Done starting upload Jul 29 18:09:13.411448449 (25h23m25.887386914s) [upload=24302]: Uploading segments... Jul 29 18:09:13.411584579 (25h23m25.887523105s) [upload=24302]: Got next segment. Inline: false Jul 29 18:09:13.411619847 (25h23m25.887558312s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:09:13.411624886 (25h23m25.887563351s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:09:13.710397165 (25h23m26.186335641s) [seg_pos={0 0}, upload=24302]: Begin upload segment... Jul 29 18:09:13.710437141 (25h23m26.186375607s) [seg_pos={0 0}, upload=24302]: Done begin upload segment. Jul 29 18:09:13.710454736 (25h23m26.186393201s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710487177 (25h23m26.186425642s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710815896 (25h23m26.186754361s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.710946497 (25h23m26.186884962s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711074761 (25h23m26.187013226s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711263392 (25h23m26.187201857s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711364526 (25h23m26.187302991s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711571492 (25h23m26.187509957s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711692223 (25h23m26.187630698s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711842250 (25h23m26.187780715s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.711924327 (25h23m26.187862792s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712130702 (25h23m26.188069167s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712384027 (25h23m26.188322492s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712664794 (25h23m26.188603269s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.712933458 (25h23m26.188871923s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:13.713218333 (25h23m26.189156798s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Uploading piece... Jul 29 18:09:15.846628636 (25h23m28.322567101s) [upload=24302]: Next returned nil segment Jul 29 18:09:15.846643754 (25h23m28.32258221s) [upload=24302]: Waiting for error group managing segments... Jul 29 18:09:16.162057685 (25h23m28.63799615s) [noise=true , address=45.91.116.10:4081 , piece_id=DGV2QCQWDJNB6464AIIHUNODKAJTWRIN6WNVE3YJWGAKM7S6ZQJQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.375105846 (25h23m28.851044301s) [noise=true , address=77.90.31.17:4771 , piece_id=GEQ3ALAZHUYMATD6OE6VU5CVWXZXQGNC3HCNGNW2EDSX5KPPQ6UQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.646830563 (25h23m29.122769028s) [noise=true , address=45.91.116.4:4939 , piece_id=4FIYQZOA2RWW7JI66P3CXEMCLI3JPBOANHGHVQI5MS7T3A2JL75A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.727683722 (25h23m29.203622187s) [noise=true , address=77.90.31.10:4517 , piece_id=6GWSQXOSB65ET7IJMGOZ4E2UDAGROC6GVCA7P64HUYXKRSHPHARQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.832472808 (25h23m29.308411273s) [noise=true , address=77.90.31.17:4772 , piece_id=CAEUN2OQKQUAVI6CKAZWFHU7LYT2RJ66R4QNRPFUU2ZQISB6G52A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:16.899504592 (25h23m29.375443057s) [noise=true , address=45.91.116.12:4225 , piece_id=IYVWIYYWZJUOIU6W6AF4PWAG3J4JSEQMQQRME3KH43Z4A2TH2ROA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.026464017 (25h23m29.502402482s) [noise=true , address=45.91.116.12:4227 , piece_id=SK47FASGK22CBBUHOZLCFVGLHB2FMOQ56WCVKXRNW2TWRXTIBLKA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.067504079 (25h23m29.543442534s) [noise=true , address=45.91.116.7:4009 , piece_id=QSQH2DPGJI2AC3A3JUPNBSCAFW5EHH2TR7O4WXO3LRJEVGH5L6HA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.108526559 (25h23m29.584465024s) [noise=true , address=45.91.116.9:4263 , piece_id=XK6UD776TJBFULO5AXI6K2SM3AY4NCBJXPUQAIOO5OKSGLPJPN4A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.122453906 (25h23m29.598392361s) [noise=true , address=45.91.116.13:4155 , piece_id=6HFG5PEDLN2G4F67VWCTF7C6754OI5P3PM6P5RS6W4SNS6EIZKIQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.251280624 (25h23m29.727219089s) [noise=true , address=77.90.31.19:4843 , piece_id=K3EYVAZ35IEJOG6CYSZ5IMQFAAAHL6VBCKEBIAL5BO3CD7FFPGAA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.276243616 (25h23m29.752182071s) [noise=true , address=77.90.31.11:4553 , piece_id=IRO7MN4IZT6WFQUM2CWUM47Q73ZH23Y6RREZFE6AX7LPGZM5FQMQ, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434347772 (25h23m29.910286227s) [noise=true , address=77.90.31.16:4734 , piece_id=K43GBYIBBWH7MXNDNOB2ELBBWH5AGUKSMXDLXHO6J3XZNERDALYA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: <nil> Jul 29 18:09:17.434373681 (25h23m29.910312146s) [seg_pos={0 0}, upload=24302]: Segment reached optimal threshold of 13 pieces. Jul 29 18:09:17.434478322 (25h23m29.910416777s) [noise=true , address=104.36.82.16:4295 , piece_id=T7SQ6LID2YBECXLOK4JVA6R3ZUJLKCF7UULSOVFQNIMOTIZ6E3TA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434577681 (25h23m29.910516146s) [noise=true , address=77.90.31.12:4589 , piece_id=GLQ5FCTTJYQPOJPGUUI6GXQAGJNHQ3E4RUYWC24MEIAGBPSIKV7A, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jul 29 18:09:17.434605815 (25h23m29.91054428s) [noise=true , address=45.91.116.10:4083 , piece_id=UTOVFQMF5CWGHIB4GWVEXT535NK72E2DJSW2N6B6YB3GUAILSYSA, seg_pos={0 0}, upload=24302]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37424->45.91.116.10:4083: use of closed network connection Jul 29 18:09:17.434647275 (25h23m29.91058574s) [seg_pos={0 0}, upload=24302]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:09:17.434722379 (25h23m29.910660844s) [upload=24302]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:09:17.434726727 (25h23m29.910665192s) [upload=24302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:09:17.731175941 (25h23m30.207114406s) [upload=24302]: Done uploading segments... Jul 29 18:09:17.731194988 (25h23m30.207133453s) [upload=24302]: Upload finished. err: <nil> Duration: 30.002286531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........................X..........X.XX......XX.XX.XXXXX.X] duration: 30.002414817s FAIL (2025-07-29 18:09:40 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002425036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.001948841s FAIL (2025-07-29 18:09:12 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== LI5DITGX5A, seg_pos={0 0}, upload=24272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Jul 29 18:07:38.131675025 (25h21m50.60761349s) [seg_pos={0 0}, upload=24272]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:07:38.131764316 (25h21m50.607702791s) [upload=24272]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:07:38.131772401 (25h21m50.607710936s) [upload=24272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:07:38.527924890 (25h21m51.003863355s) [upload=24272]: Done uploading segments... Jul 29 18:07:38.527942053 (25h21m51.003880518s) [upload=24272]: Upload finished. err: <nil> Jul 29 18:08:43.176330729 (25h22m55.652269194s) [upload=24289]: Starting upload Jul 29 18:08:43.176411805 (25h22m55.65235028s) [upload=24289]: Done starting upload Jul 29 18:08:43.176489834 (25h22m55.652428299s) [upload=24289]: Uploading segments... Jul 29 18:08:43.176707140 (25h22m55.652645605s) [upload=24289]: Got next segment. Inline: false Jul 29 18:08:43.176756394 (25h22m55.652694859s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:08:43.176762135 (25h22m55.6527006s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:08:43.412967323 (25h22m55.888905798s) [seg_pos={0 0}, upload=24289]: Begin upload segment... Jul 29 18:08:43.413017619 (25h22m55.888956084s) [seg_pos={0 0}, upload=24289]: Done begin upload segment. Jul 29 18:08:43.413043158 (25h22m55.888981623s) [noise=true , address=104.36.82.2:4730 , piece_id=USQQXU2LRC56BCFHP5SHIATJAR6MVDCKJZP3GIQI3SLXC2SKSBZQ, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.413113212 (25h22m55.889051687s) [noise=true , address=45.91.116.5:4297 , piece_id=OK4LQC7ZJRU6F6VEIFLQFJ4QHUZDQ3Y4T5FKYLHS7BWI6EKR6OOA, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.413774778 (25h22m55.889713243s) [noise=true , address=45.91.116.6:4973 , piece_id=UZIPYP3CVPLQ73CPG5F6XZTZHUY2SD332TSJH6IRUG76OOMVEK2Q, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.414067358 (25h22m55.890005833s) [noise=true , address=77.90.31.12:4592 , piece_id=CJFCNV6ZMTT74GE73YOCCQPHSTIKUBC34JTOORMR7M43WM7VAMSQ, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.414384334 (25h22m55.890322799s) [noise=true , address=45.91.116.6:4974 , piece_id=7WLWKTMV7EKOQRTWTMCC7KEBGTSWSM4KU2CTNCVSA4IRRTMES72Q, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.414799439 (25h22m55.890737914s) [noise=true , address=104.36.82.4:4802 , piece_id=O7KV4GKVLMBLQ6HWGD3PQ7F4QLY7W4SI2O3XOKG35XSFR6DMX3LA, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.414831941 (25h22m55.890770406s) [noise=true , address=77.90.31.18:4806 , piece_id=S445CBWVJFOKCKKDHG2OGRYV3HLX6CHOVEOWCUAJ4MCE4GRUDC2A, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.415348690 (25h22m55.891287155s) [noise=true , address=45.91.116.14:4191 , piece_id=PDBJX5Y6CW6YDGABSBFRRLI6M4NARW6OFV735MTWCIEKBO33AHQA, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.415459582 (25h22m55.891398037s) [noise=true , address=77.90.31.15:4697 , piece_id=2LPJKMPGCJFOL3HU3GEE5BLVHGMOJFUWQZSAXSG4XH6PRJZQHVXA, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.415751620 (25h22m55.891690085s) [noise=true , address=45.91.116.7:4009 , piece_id=I4CDOU74CTLWYF2KJM6YTRTFFPHPOP4EHGEK6OXKOEYNTKB2CHNQ, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.416152437 (25h22m55.892090902s) [noise=true , address=45.91.116.9:4262 , piece_id=FYDK3HYJURXOC3ZH43WL6M6JMLZ52MUY5G5LZIYSA4ZNIJEWUQGA, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.416417304 (25h22m55.892355779s) [noise=true , address=77.90.31.13:4628 , piece_id=GM72HJDK5MK4IWKYUF2WHUVV6RLUQXW5FK4OY4HH52RDHKQEXLJQ, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.416655029 (25h22m55.892593494s) [noise=true , address=104.36.82.13:4185 , piece_id=ZMJFBGCMTG4RW7LFTR6TFJLEC6YGZ3YAOHQFJXOK33O2UCVFV6QA, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.416980752 (25h22m55.892919217s) [noise=true , address=45.91.116.8:4047 , piece_id=MCV2DY4YXPKYDTVEALVBYTZILK4MXA23GJPSTTLJ2CFPFA7LPLRA, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.417452715 (25h22m55.89339118s) [noise=true , address=77.90.31.21:4879 , piece_id=QARDUHDHU377NF5TF5FKKBVCUVCBKEOYV4LGF6KQWB6K6F7IDEGQ, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:43.417851949 (25h22m55.893790414s) [noise=true , address=77.90.31.14:4662 , piece_id=522FHIDDDCZOXN7VWE45UXL6GH5WDQPYJR7HMIUT75PKHFQYMXGA, seg_pos={0 0}, upload=24289]: Uploading piece... Jul 29 18:08:45.344695291 (25h22m57.820633756s) [upload=24289]: Next returned nil segment Jul 29 18:08:45.344717584 (25h22m57.820656059s) [upload=24289]: Waiting for error group managing segments... Jul 29 18:08:45.861617763 (25h22m58.337556228s) [noise=true , address=77.90.31.15:4697 , piece_id=2LPJKMPGCJFOL3HU3GEE5BLVHGMOJFUWQZSAXSG4XH6PRJZQHVXA, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.864578157 (25h22m58.340516622s) [noise=true , address=45.91.116.6:4973 , piece_id=UZIPYP3CVPLQ73CPG5F6XZTZHUY2SD332TSJH6IRUG76OOMVEK2Q, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.865374040 (25h22m58.341312505s) [noise=true , address=77.90.31.14:4662 , piece_id=522FHIDDDCZOXN7VWE45UXL6GH5WDQPYJR7HMIUT75PKHFQYMXGA, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.867053393 (25h22m58.342991858s) [noise=true , address=104.36.82.4:4802 , piece_id=O7KV4GKVLMBLQ6HWGD3PQ7F4QLY7W4SI2O3XOKG35XSFR6DMX3LA, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.868739779 (25h22m58.344678244s) [noise=true , address=45.91.116.5:4297 , piece_id=OK4LQC7ZJRU6F6VEIFLQFJ4QHUZDQ3Y4T5FKYLHS7BWI6EKR6OOA, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.870315044 (25h22m58.346253509s) [noise=true , address=104.36.82.13:4185 , piece_id=ZMJFBGCMTG4RW7LFTR6TFJLEC6YGZ3YAOHQFJXOK33O2UCVFV6QA, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.873506260 (25h22m58.349444725s) [noise=true , address=45.91.116.9:4262 , piece_id=FYDK3HYJURXOC3ZH43WL6M6JMLZ52MUY5G5LZIYSA4ZNIJEWUQGA, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.883031053 (25h22m58.358969518s) [noise=true , address=77.90.31.21:4879 , piece_id=QARDUHDHU377NF5TF5FKKBVCUVCBKEOYV4LGF6KQWB6K6F7IDEGQ, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.885896446 (25h22m58.361834941s) [noise=true , address=77.90.31.13:4628 , piece_id=GM72HJDK5MK4IWKYUF2WHUVV6RLUQXW5FK4OY4HH52RDHKQEXLJQ, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.891393664 (25h22m58.367332129s) [noise=true , address=104.36.82.2:4730 , piece_id=USQQXU2LRC56BCFHP5SHIATJAR6MVDCKJZP3GIQI3SLXC2SKSBZQ, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.891799060 (25h22m58.367737525s) [noise=true , address=45.91.116.7:4009 , piece_id=I4CDOU74CTLWYF2KJM6YTRTFFPHPOP4EHGEK6OXKOEYNTKB2CHNQ, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.907850131 (25h22m58.383788596s) [noise=true , address=77.90.31.18:4806 , piece_id=S445CBWVJFOKCKKDHG2OGRYV3HLX6CHOVEOWCUAJ4MCE4GRUDC2A, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.931531210 (25h22m58.407469675s) [noise=true , address=45.91.116.6:4974 , piece_id=7WLWKTMV7EKOQRTWTMCC7KEBGTSWSM4KU2CTNCVSA4IRRTMES72Q, seg_pos={0 0}, upload=24289]: Done uploading piece. err: <nil> Jul 29 18:08:45.931549515 (25h22m58.40748798s) [seg_pos={0 0}, upload=24289]: Segment reached optimal threshold of 13 pieces. Jul 29 18:08:45.931656049 (25h22m58.407594514s) [noise=true , address=45.91.116.8:4047 , piece_id=MCV2DY4YXPKYDTVEALVBYTZILK4MXA23GJPSTTLJ2CFPFA7LPLRA, seg_pos={0 0}, upload=24289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 29 18:08:45.931865430 (25h22m58.407803895s) [noise=true , address=45.91.116.14:4191 , piece_id=PDBJX5Y6CW6YDGABSBFRRLI6M4NARW6OFV735MTWCIEKBO33AHQA, seg_pos={0 0}, upload=24289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Jul 29 18:08:45.931899935 (25h22m58.4078384s) [noise=true , address=77.90.31.12:4592 , piece_id=CJFCNV6ZMTT74GE73YOCCQPHSTIKUBC34JTOORMR7M43WM7VAMSQ, seg_pos={0 0}, upload=24289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54518->77.90.31.12:4592: use of closed network connection Jul 29 18:08:45.931909684 (25h22m58.407848149s) [seg_pos={0 0}, upload=24289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:08:45.931964368 (25h22m58.407902833s) [upload=24289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:08:45.931968146 (25h22m58.407906612s) [upload=24289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:08:46.337748236 (25h22m58.813686701s) [upload=24289]: Done uploading segments... Jul 29 18:08:46.337766691 (25h22m58.813705156s) [upload=24289]: Upload finished. err: <nil> Duration: 30.00196385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................X..........X.XX......XX.XX.XXXXX.] duration: 25.820185069s FAIL (2025-07-29 18:09:09 UTC) orbiter warning: uplink: context deadline exceeded Duration: 25.820228952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X.................................X.......................] duration: 1.960507374s FAIL (2025-07-29 18:09:06 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.960530518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.................................X......................] duration: 2.056394526s FAIL (2025-07-29 18:08:44 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.05641751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................X..........X.XX......XX.XX.XXXXX] duration: 30.002206798s FAIL (2025-07-29 18:08:07 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002240493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 30.00262489s FAIL (2025-07-29 18:08:03 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== HGQOX5IA, seg_pos={0 0}, upload=24257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:38698->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:38698->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38698->77.90.31.17:4771: use of closed network connection Jul 29 18:06:39.897868413 (25h20m52.373806878s) [seg_pos={0 0}, upload=24257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:06:39.897928167 (25h20m52.373866622s) [upload=24257]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:06:39.897932355 (25h20m52.37387082s) [upload=24257]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:06:40.217925898 (25h20m52.693864363s) [upload=24257]: Done uploading segments... Jul 29 18:06:40.217945646 (25h20m52.693884111s) [upload=24257]: Upload finished. err: <nil> Jul 29 18:07:34.344743998 (25h21m46.820682463s) [upload=24272]: Starting upload Jul 29 18:07:34.344806167 (25h21m46.820744632s) [upload=24272]: Done starting upload Jul 29 18:07:34.344910767 (25h21m46.820849232s) [upload=24272]: Uploading segments... Jul 29 18:07:34.345056375 (25h21m46.82099484s) [upload=24272]: Got next segment. Inline: false Jul 29 18:07:34.345105139 (25h21m46.821043604s) [upload=24272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:07:34.345110519 (25h21m46.821048984s) [upload=24272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:07:34.579006258 (25h21m47.054944733s) [seg_pos={0 0}, upload=24272]: Begin upload segment... Jul 29 18:07:34.579065792 (25h21m47.055004257s) [seg_pos={0 0}, upload=24272]: Done begin upload segment. Jul 29 18:07:34.579097052 (25h21m47.055035587s) [noise=true , address=104.36.82.9:4007 , piece_id=2LCDSDH3IUVYD5SJSLOTZQOLLNPSECMO3O4Q5OV3TGOIG7ZYD42A, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.579113183 (25h21m47.055051648s) [noise=true , address=45.91.116.8:4045 , piece_id=LJTTWKYFASEMBFWJWHH7FUSDCWJUFI5U7KWRFV6BCQISN7LZPC5A, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.579506736 (25h21m47.055445201s) [noise=true , address=45.91.116.4:4939 , piece_id=JAHLJHUORHUPZLG2LOGP4Z3OFG3I75PA2J2VKE5LDJLCCPRN3INQ, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.579851314 (25h21m47.055789891s) [noise=true , address=104.36.82.3:4767 , piece_id=OMJAUHO2MAUNIU7TM4YGHOGMV2U46YHFHWAATDTRHEFWPTOQBQCA, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.579883417 (25h21m47.055821892s) [noise=true , address=77.90.31.12:4592 , piece_id=4C2MDIUEEFR3PZZURNHK2O3COZ4VMA5U7C34RRFPEXSY3FYPC4YA, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.580167289 (25h21m47.056105754s) [noise=true , address=45.91.116.12:4227 , piece_id=WOYY3Q65JFFK4J4EWK5SROCEG7TMTKXXBB2L3XISTDUBPAPWM2DA, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.580496199 (25h21m47.056434654s) [noise=true , address=104.36.82.10:4078 , piece_id=4OXJDRMM2SOSXNJZQFMO7GBTK6JS5ALKRD26VNRHPRJE2JUZNBBA, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.580565962 (25h21m47.056504427s) [noise=true , address=45.91.116.5:4299 , piece_id=TL4KTFH24S66P7FCY2OAOQYNEMM5ZMZ3AU7EO3BYZRSMLM6T2OEA, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.580927134 (25h21m47.056865599s) [noise=true , address=77.90.31.14:4661 , piece_id=7RWX4FZLWOTXED3RDWV23GAZVVDMT7V6SU7DSSXWMPVTJRYJJCWQ, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.580999061 (25h21m47.056937586s) [noise=true , address=77.90.31.15:4698 , piece_id=3Y2IN5BWWHQJ2GUAXOS6JJAQHDIPPAQETXM7JJQ5LUGO6BZQDNCA, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.581153386 (25h21m47.057091851s) [noise=true , address=45.91.116.8:4048 , piece_id=Y37IUUSCI63OLTXTJSRM2LFKMOLNN6TO4OYFZWGJ3X3IYPWRIJOA, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.581424705 (25h21m47.057363171s) [noise=true , address=45.91.116.6:4974 , piece_id=ZYWC63UUKBBIZWPXU4CAZZD743TTMIZ4LK6SJW44TXFRRR4PKY2A, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.581484249 (25h21m47.057422714s) [noise=true , address=45.91.116.10:4082 , piece_id=DDHOMBSX46FZKVCUV7PKPGAXVP5OQP5NA34YFMWXGP2CXUNBKCEQ, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.581985839 (25h21m47.057924304s) [noise=true , address=45.91.116.11:4118 , piece_id=TAEP2S56OREJQADGWXSNJAWIU6XPCFVPXF7C36JVW6ZD7HYHWJSA, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.582348332 (25h21m47.058286797s) [noise=true , address=45.91.116.4:4938 , piece_id=IZ3HD3AUXK7QQ2T6DSPGFHO57J72WK33XCSM3Z7EJTLI5DITGX5A, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:34.582738769 (25h21m47.058677234s) [noise=true , address=77.90.31.11:4555 , piece_id=UNCWE4CUFJCMORHVYATNLAH3BTBZK4CQWGYYRGJDZSKDJQF6XIJQ, seg_pos={0 0}, upload=24272]: Uploading piece... Jul 29 18:07:36.566279180 (25h21m49.042217645s) [upload=24272]: Next returned nil segment Jul 29 18:07:36.566303376 (25h21m49.042241841s) [upload=24272]: Waiting for error group managing segments... Jul 29 18:07:36.916306617 (25h21m49.392245092s) [noise=true , address=45.91.116.12:4227 , piece_id=WOYY3Q65JFFK4J4EWK5SROCEG7TMTKXXBB2L3XISTDUBPAPWM2DA, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:36.922841050 (25h21m49.398779515s) [noise=true , address=45.91.116.6:4974 , piece_id=ZYWC63UUKBBIZWPXU4CAZZD743TTMIZ4LK6SJW44TXFRRR4PKY2A, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:36.956263768 (25h21m49.432202233s) [noise=true , address=45.91.116.8:4048 , piece_id=Y37IUUSCI63OLTXTJSRM2LFKMOLNN6TO4OYFZWGJ3X3IYPWRIJOA, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:36.970650575 (25h21m49.44658904s) [noise=true , address=77.90.31.11:4555 , piece_id=UNCWE4CUFJCMORHVYATNLAH3BTBZK4CQWGYYRGJDZSKDJQF6XIJQ, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:36.981901101 (25h21m49.457839566s) [noise=true , address=45.91.116.10:4082 , piece_id=DDHOMBSX46FZKVCUV7PKPGAXVP5OQP5NA34YFMWXGP2CXUNBKCEQ, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:36.992615501 (25h21m49.468553966s) [noise=true , address=45.91.116.8:4045 , piece_id=LJTTWKYFASEMBFWJWHH7FUSDCWJUFI5U7KWRFV6BCQISN7LZPC5A, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:36.997786975 (25h21m49.47372544s) [noise=true , address=77.90.31.14:4661 , piece_id=7RWX4FZLWOTXED3RDWV23GAZVVDMT7V6SU7DSSXWMPVTJRYJJCWQ, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:37.009347394 (25h21m49.48528586s) [noise=true , address=77.90.31.15:4698 , piece_id=3Y2IN5BWWHQJ2GUAXOS6JJAQHDIPPAQETXM7JJQ5LUGO6BZQDNCA, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:37.027882629 (25h21m49.503821094s) [noise=true , address=104.36.82.3:4767 , piece_id=OMJAUHO2MAUNIU7TM4YGHOGMV2U46YHFHWAATDTRHEFWPTOQBQCA, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:37.033453617 (25h21m49.509392082s) [noise=true , address=77.90.31.12:4592 , piece_id=4C2MDIUEEFR3PZZURNHK2O3COZ4VMA5U7C34RRFPEXSY3FYPC4YA, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:37.404395492 (25h21m49.880333967s) [noise=true , address=104.36.82.9:4007 , piece_id=2LCDSDH3IUVYD5SJSLOTZQOLLNPSECMO3O4Q5OV3TGOIG7ZYD42A, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:37.407780949 (25h21m49.883719414s) [noise=true , address=45.91.116.11:4118 , piece_id=TAEP2S56OREJQADGWXSNJAWIU6XPCFVPXF7C36JVW6ZD7HYHWJSA, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:38.131413775 (25h21m50.60735223s) [noise=true , address=104.36.82.10:4078 , piece_id=4OXJDRMM2SOSXNJZQFMO7GBTK6JS5ALKRD26VNRHPRJE2JUZNBBA, seg_pos={0 0}, upload=24272]: Done uploading piece. err: <nil> Jul 29 18:07:38.131436769 (25h21m50.607375234s) [seg_pos={0 0}, upload=24272]: Segment reached optimal threshold of 13 pieces. Jul 29 18:07:38.131521371 (25h21m50.607459836s) [noise=true , address=45.91.116.4:4939 , piece_id=JAHLJHUORHUPZLG2LOGP4Z3OFG3I75PA2J2VKE5LDJLCCPRN3INQ, seg_pos={0 0}, upload=24272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 29 18:07:38.131578300 (25h21m50.607516755s) [noise=true , address=45.91.116.5:4299 , piece_id=TL4KTFH24S66P7FCY2OAOQYNEMM5ZMZ3AU7EO3BYZRSMLM6T2OEA, seg_pos={0 0}, upload=24272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jul 29 18:07:38.131638034 (25h21m50.607576499s) [noise=true , address=45.91.116.4:4938 , piece_id=IZ3HD3AUXK7QQ2T6DSPGFHO57J72WK33XCSM3Z7EJTLI5DITGX5A, seg_pos={0 0}, upload=24272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Jul 29 18:07:38.131675025 (25h21m50.60761349s) [seg_pos={0 0}, upload=24272]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:07:38.131764316 (25h21m50.607702791s) [upload=24272]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:07:38.131772401 (25h21m50.607710936s) [upload=24272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:07:38.527924890 (25h21m51.003863355s) [upload=24272]: Done uploading segments... Jul 29 18:07:38.527942053 (25h21m51.003880518s) [upload=24272]: Upload finished. err: <nil> Duration: 30.002653325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X...X.................................X.....................] duration: 1.85602361s FAIL (2025-07-29 18:07:34 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.856053047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............................X..........X.XX......XX.XX.XXXX] duration: 30.001624606s FAIL (2025-07-29 18:07:30 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.001641388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 30.002347327s FAIL (2025-07-29 18:07:05 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7D7A, seg_pos={0 0}, upload=24242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Jul 29 18:05:32.502914306 (25h19m44.978852781s) [seg_pos={0 0}, upload=24242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:05:32.503010881 (25h19m44.978949346s) [upload=24242]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:05:32.503016191 (25h19m44.978954656s) [upload=24242]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:05:32.972533130 (25h19m45.448471595s) [upload=24242]: Done uploading segments... Jul 29 18:05:32.972551957 (25h19m45.448490412s) [upload=24242]: Upload finished. err: <nil> Jul 29 18:06:36.236477840 (25h20m48.712416295s) [upload=24257]: Starting upload Jul 29 18:06:36.236541663 (25h20m48.712480128s) [upload=24257]: Done starting upload Jul 29 18:06:36.236621286 (25h20m48.712559751s) [upload=24257]: Uploading segments... Jul 29 18:06:36.236713161 (25h20m48.712651616s) [upload=24257]: Got next segment. Inline: false Jul 29 18:06:36.236740634 (25h20m48.712679089s) [upload=24257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:06:36.236745533 (25h20m48.712683998s) [upload=24257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:06:36.471175704 (25h20m48.947114179s) [seg_pos={0 0}, upload=24257]: Begin upload segment... Jul 29 18:06:36.471225810 (25h20m48.947164275s) [seg_pos={0 0}, upload=24257]: Done begin upload segment. Jul 29 18:06:36.471252301 (25h20m48.947190766s) [noise=true , address=104.36.82.4:4803 , piece_id=WI6CTRFPPKKURBBYUD47DRR3AXW6QUGVTXGFDPRCMI7TSVX63S7Q, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.471264154 (25h20m48.947202649s) [noise=true , address=77.90.31.19:4842 , piece_id=AACEXR2DNEAYHMMZZF2GWG5JHPA5IDNXN6HUMC3GCJOYKMBZVD3A, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.472012606 (25h20m48.947951071s) [noise=true , address=104.36.82.3:4765 , piece_id=F6E33TXMTGCK2PGU2ZSAN3J4OREMJL4C4XPGIZD2MAHQ7F2CLBXA, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.472055147 (25h20m48.947993612s) [noise=true , address=104.36.82.8:4969 , piece_id=L2ZDVPDYSOFFWGBGED36GUJW5KCZHLJC2TOUTJNHX2YQI62SUQPA, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.472520277 (25h20m48.948458742s) [noise=true , address=45.91.116.7:4010 , piece_id=XPZ7HZBYWTNQXAFCWVKGSZSDACF36HRN7VH372FVNIBNSFBHIQDA, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.472715941 (25h20m48.948654426s) [noise=true , address=45.91.116.14:4190 , piece_id=F6QYQXCVPSHR3U6IEDV2YKRZVARRCSWN7XY6H43JDBODZLNF573Q, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.473196491 (25h20m48.949134956s) [noise=true , address=77.90.31.15:4699 , piece_id=OO374CN77B5FRU2ESM5NZ33BMBWGK5F7WMQC2KLML54S6INARSXA, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.473400060 (25h20m48.949338525s) [noise=true , address=45.91.116.12:4225 , piece_id=EEMTHZDUIRPN7OCPVUAMFP64RNA6DFJM5BEJZCMJJ3PACJG5366Q, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.473742706 (25h20m48.949681171s) [noise=true , address=104.36.82.7:4910 , piece_id=TGF565AIDFOS4X4IBBJKFFL3IORQ3UUXHHPJISNIYIGE56QKRWDA, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.474139014 (25h20m48.950077479s) [noise=true , address=45.91.116.8:4048 , piece_id=SBOST33QT6VQELH6RA2KQ5N6INJ4TXSE3VGTEIGTLUWHOIZOHTVA, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.474296766 (25h20m48.950235231s) [noise=true , address=104.36.82.13:4187 , piece_id=COZZK2U5546JHLOV5CX3XGM46G3UABHC72KDDM6TX5FWZN5QPM5A, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.474893238 (25h20m48.950831703s) [noise=true , address=45.91.116.4:4939 , piece_id=5WC7UI6M5DHWSSOQW6YMIYDCOCY2FCRPSV2LF67MMRMLWSCGWJ3A, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.475305627 (25h20m48.951244092s) [noise=true , address=77.90.31.17:4771 , piece_id=2MADD636CESSFZSZAR4YO6T2LVYD4ZDOL3KG3XS5HII3HGQOX5IA, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.475602775 (25h20m48.95154123s) [noise=true , address=45.91.116.10:4082 , piece_id=7LN4IIE6DZT4WKYNR62XVNNHLZS46KY4NNNAA7MDBPR7GMPE72CQ, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.476239343 (25h20m48.952177818s) [noise=true , address=45.91.116.13:4154 , piece_id=KQSGS2HSJNLFVNJVPZW7GRDZY2WQUMNOC2IJ2ZSSSKWW3KGZMK7Q, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:36.476419237 (25h20m48.952357702s) [noise=true , address=104.36.82.10:4079 , piece_id=AM6B4OBM2R3OATNSTDMLVIAULZOIUIBVSDCWQ6L2GIQJNNJJ2WQQ, seg_pos={0 0}, upload=24257]: Uploading piece... Jul 29 18:06:38.459026757 (25h20m50.934965212s) [upload=24257]: Next returned nil segment Jul 29 18:06:38.459044211 (25h20m50.934982676s) [upload=24257]: Waiting for error group managing segments... Jul 29 18:06:38.778736237 (25h20m51.254674702s) [noise=true , address=45.91.116.10:4082 , piece_id=7LN4IIE6DZT4WKYNR62XVNNHLZS46KY4NNNAA7MDBPR7GMPE72CQ, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:38.852600434 (25h20m51.328538899s) [noise=true , address=104.36.82.8:4969 , piece_id=L2ZDVPDYSOFFWGBGED36GUJW5KCZHLJC2TOUTJNHX2YQI62SUQPA, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:38.910743861 (25h20m51.386682326s) [noise=true , address=45.91.116.12:4225 , piece_id=EEMTHZDUIRPN7OCPVUAMFP64RNA6DFJM5BEJZCMJJ3PACJG5366Q, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:38.961633589 (25h20m51.437572054s) [noise=true , address=77.90.31.15:4699 , piece_id=OO374CN77B5FRU2ESM5NZ33BMBWGK5F7WMQC2KLML54S6INARSXA, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.349348123 (25h20m51.825286588s) [noise=true , address=45.91.116.14:4190 , piece_id=F6QYQXCVPSHR3U6IEDV2YKRZVARRCSWN7XY6H43JDBODZLNF573Q, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.431545083 (25h20m51.907483548s) [noise=true , address=77.90.31.19:4842 , piece_id=AACEXR2DNEAYHMMZZF2GWG5JHPA5IDNXN6HUMC3GCJOYKMBZVD3A, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.838843368 (25h20m52.314781833s) [noise=true , address=104.36.82.7:4910 , piece_id=TGF565AIDFOS4X4IBBJKFFL3IORQ3UUXHHPJISNIYIGE56QKRWDA, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.839771604 (25h20m52.315710069s) [noise=true , address=45.91.116.4:4939 , piece_id=5WC7UI6M5DHWSSOQW6YMIYDCOCY2FCRPSV2LF67MMRMLWSCGWJ3A, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.840788550 (25h20m52.316727025s) [noise=true , address=104.36.82.10:4079 , piece_id=AM6B4OBM2R3OATNSTDMLVIAULZOIUIBVSDCWQ6L2GIQJNNJJ2WQQ, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.879349981 (25h20m52.355288446s) [noise=true , address=45.91.116.7:4010 , piece_id=XPZ7HZBYWTNQXAFCWVKGSZSDACF36HRN7VH372FVNIBNSFBHIQDA, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.893429960 (25h20m52.369368425s) [noise=true , address=45.91.116.13:4154 , piece_id=KQSGS2HSJNLFVNJVPZW7GRDZY2WQUMNOC2IJ2ZSSSKWW3KGZMK7Q, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.893511276 (25h20m52.369449741s) [noise=true , address=104.36.82.3:4765 , piece_id=F6E33TXMTGCK2PGU2ZSAN3J4OREMJL4C4XPGIZD2MAHQ7F2CLBXA, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.894046510 (25h20m52.369984975s) [noise=true , address=104.36.82.4:4803 , piece_id=WI6CTRFPPKKURBBYUD47DRR3AXW6QUGVTXGFDPRCMI7TSVX63S7Q, seg_pos={0 0}, upload=24257]: Done uploading piece. err: <nil> Jul 29 18:06:39.894055086 (25h20m52.369993551s) [seg_pos={0 0}, upload=24257]: Segment reached optimal threshold of 13 pieces. Jul 29 18:06:39.897644394 (25h20m52.373582859s) [noise=true , address=104.36.82.13:4187 , piece_id=COZZK2U5546JHLOV5CX3XGM46G3UABHC72KDDM6TX5FWZN5QPM5A, seg_pos={0 0}, upload=24257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Jul 29 18:06:39.897731671 (25h20m52.373670136s) [noise=true , address=45.91.116.8:4048 , piece_id=SBOST33QT6VQELH6RA2KQ5N6INJ4TXSE3VGTEIGTLUWHOIZOHTVA, seg_pos={0 0}, upload=24257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Jul 29 18:06:39.897857271 (25h20m52.373795736s) [noise=true , address=77.90.31.17:4771 , piece_id=2MADD636CESSFZSZAR4YO6T2LVYD4ZDOL3KG3XS5HII3HGQOX5IA, seg_pos={0 0}, upload=24257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:38698->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:38698->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38698->77.90.31.17:4771: use of closed network connection Jul 29 18:06:39.897868413 (25h20m52.373806878s) [seg_pos={0 0}, upload=24257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:06:39.897928167 (25h20m52.373866622s) [upload=24257]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:06:39.897932355 (25h20m52.37387082s) [upload=24257]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:06:40.217925898 (25h20m52.693864363s) [upload=24257]: Done uploading segments... Jul 29 18:06:40.217945646 (25h20m52.693884111s) [upload=24257]: Upload finished. err: <nil> Duration: 30.002365943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X...X.................................X....................] duration: 1.969964156s FAIL (2025-07-29 18:07:04 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.969989605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............................X..........X.XX......XX.XX.XXX] duration: 30.002243303s FAIL (2025-07-29 18:07:00 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002265774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X...X.................................X...................] duration: 2.101716616s FAIL (2025-07-29 18:06:25 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.101748406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [...............................X..........X.XX......XX.XX.XX] duration: 30.002024866s FAIL (2025-07-29 18:06:20 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002046057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 30.001920461s FAIL (2025-07-29 18:05:58 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== IAYHKNHQ, seg_pos={0 0}, upload=24231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:56648->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:56648->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56648->45.91.116.11:4118: use of closed network connection Jul 29 18:04:41.043389960 (25h18m53.519328425s) [seg_pos={0 0}, upload=24231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:04:41.043545126 (25h18m53.519483591s) [upload=24231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:04:41.043550778 (25h18m53.519489243s) [upload=24231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:04:41.465788927 (25h18m53.941727392s) [upload=24231]: Done uploading segments... Jul 29 18:04:41.465842159 (25h18m53.941780624s) [upload=24231]: Upload finished. err: <nil> Jul 29 18:05:28.986432727 (25h19m41.462371192s) [upload=24242]: Starting upload Jul 29 18:05:28.986491639 (25h19m41.462430094s) [upload=24242]: Done starting upload Jul 29 18:05:28.986542938 (25h19m41.462481403s) [upload=24242]: Uploading segments... Jul 29 18:05:28.986757388 (25h19m41.462695853s) [upload=24242]: Got next segment. Inline: false Jul 29 18:05:28.986783709 (25h19m41.462722174s) [upload=24242]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:05:28.986788207 (25h19m41.462726672s) [upload=24242]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:05:29.249497366 (25h19m41.725435821s) [seg_pos={0 0}, upload=24242]: Begin upload segment... Jul 29 18:05:29.249539306 (25h19m41.725477771s) [seg_pos={0 0}, upload=24242]: Done begin upload segment. Jul 29 18:05:29.249564123 (25h19m41.725502588s) [noise=true , address=77.90.31.13:4627 , piece_id=KEOGIW5FWFCQKECJBNJ6O5DLJSTFBQOAMQZGNOSGTWJ4KDRH3UOA, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.249604131 (25h19m41.725542586s) [noise=true , address=77.90.31.16:4733 , piece_id=BBSO4JNO6QS7LBDGP7G7C675IQBBDJTXXNG3QS25G2FKFWDEZFQA, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.249941085 (25h19m41.72587955s) [noise=true , address=45.91.116.12:4225 , piece_id=VMGZUD5SOKDESZSQK3BAMHJG27GOAEFY2BOMRAPJ3NVKXTUUPDLA, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.249948940 (25h19m41.725887395s) [noise=true , address=77.90.31.11:4553 , piece_id=FVJHS7V5GKQUR4DEJXKEMHBPJM6MYAWFPDC7JCQXJEE5KFLXJC3Q, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.250202085 (25h19m41.72614055s) [noise=true , address=45.91.116.11:4118 , piece_id=GK4JF3RXPI4XWWYSDEODW2RERTHWBP6ENPTLALTB6WNTVFMDRVTQ, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.250214188 (25h19m41.726152643s) [noise=true , address=45.91.116.8:4047 , piece_id=5TM453FATCY2NHMTESTOAR4ZYOBD3WNAKYACRMMQ34EWJZUEZPSQ, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.250453075 (25h19m41.72639154s) [noise=true , address=104.36.82.9:4007 , piece_id=3ZBOLLCN6A7YQ2UIY3OU5NUGOUGVSBANC6SY2MYIK3VADF3QOLVQ, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.250466360 (25h19m41.726404825s) [noise=true , address=104.36.82.8:4969 , piece_id=6RSYLRP2ME6OKBDANNBD77TZEHWABCXCI6LVXD6KDMF46AAOWQLQ, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.250753690 (25h19m41.726692155s) [noise=true , address=77.90.31.18:4806 , piece_id=SNX5YHNAOBZWR32HKHWCV4L3HDK6HOTF7SOU7WG7DSOHTZC77D7A, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.250822171 (25h19m41.726760636s) [noise=true , address=104.36.82.15:4257 , piece_id=JT4VHXLRZ3BMRY7NWW4NU6QTDV5ZISNPW3POACOKLXKTZAKGOQ2Q, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.251006444 (25h19m41.726944909s) [noise=true , address=77.90.31.17:4772 , piece_id=TPUSDMZCAEDA7K5SMKR3NM7F3SDFXRXYTPYYAXAKZMF4I4APDHHA, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.251038525 (25h19m41.72697699s) [noise=true , address=77.90.31.14:4663 , piece_id=ISXY5DWAFL6JERL3OLJF6ECHONXJYIYJFB7FCIOJXX4EV6KF2C3Q, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.251204933 (25h19m41.727143388s) [noise=true , address=104.36.82.2:4732 , piece_id=YAXB3EKNAUJ44IPWP737SGQI6W7B6O623TGRTCTAUHXSZRZJLS4A, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.251245131 (25h19m41.727183596s) [noise=true , address=45.91.116.5:4299 , piece_id=VFNHG4NHNZTRAZEHLO6H5KJU5C3QDZFLQVKLZPG27RPYOPRAJW4A, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.251499737 (25h19m41.727438202s) [noise=true , address=104.36.82.13:4187 , piece_id=7HZAS3UP55FSLSDDSD456SQ75QQRFTYBTCATCSX4L5O6VJY4I2HA, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:29.251833907 (25h19m41.727772372s) [noise=true , address=77.90.31.10:4520 , piece_id=AUZ2VUSENXBWYSSS6EW2ZD5P5FALKQTBMERBEEPNJWARS5VBLMCA, seg_pos={0 0}, upload=24242]: Uploading piece... Jul 29 18:05:31.253074765 (25h19m43.72901323s) [upload=24242]: Next returned nil segment Jul 29 18:05:31.253096656 (25h19m43.729035121s) [upload=24242]: Waiting for error group managing segments... Jul 29 18:05:31.636448074 (25h19m44.112386539s) [noise=true , address=45.91.116.12:4225 , piece_id=VMGZUD5SOKDESZSQK3BAMHJG27GOAEFY2BOMRAPJ3NVKXTUUPDLA, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:31.799471240 (25h19m44.275409705s) [noise=true , address=77.90.31.13:4627 , piece_id=KEOGIW5FWFCQKECJBNJ6O5DLJSTFBQOAMQZGNOSGTWJ4KDRH3UOA, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:31.890868011 (25h19m44.366806486s) [noise=true , address=45.91.116.8:4047 , piece_id=5TM453FATCY2NHMTESTOAR4ZYOBD3WNAKYACRMMQ34EWJZUEZPSQ, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:31.938774893 (25h19m44.414713348s) [noise=true , address=104.36.82.2:4732 , piece_id=YAXB3EKNAUJ44IPWP737SGQI6W7B6O623TGRTCTAUHXSZRZJLS4A, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:31.958293248 (25h19m44.434231703s) [noise=true , address=104.36.82.15:4257 , piece_id=JT4VHXLRZ3BMRY7NWW4NU6QTDV5ZISNPW3POACOKLXKTZAKGOQ2Q, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:31.973175653 (25h19m44.449114118s) [noise=true , address=45.91.116.5:4299 , piece_id=VFNHG4NHNZTRAZEHLO6H5KJU5C3QDZFLQVKLZPG27RPYOPRAJW4A, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:32.032954099 (25h19m44.508892564s) [noise=true , address=77.90.31.10:4520 , piece_id=AUZ2VUSENXBWYSSS6EW2ZD5P5FALKQTBMERBEEPNJWARS5VBLMCA, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:32.222196351 (25h19m44.698134816s) [noise=true , address=104.36.82.13:4187 , piece_id=7HZAS3UP55FSLSDDSD456SQ75QQRFTYBTCATCSX4L5O6VJY4I2HA, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:32.257729378 (25h19m44.733667843s) [noise=true , address=77.90.31.16:4733 , piece_id=BBSO4JNO6QS7LBDGP7G7C675IQBBDJTXXNG3QS25G2FKFWDEZFQA, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:32.259243835 (25h19m44.7351823s) [noise=true , address=45.91.116.11:4118 , piece_id=GK4JF3RXPI4XWWYSDEODW2RERTHWBP6ENPTLALTB6WNTVFMDRVTQ, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:32.431663290 (25h19m44.907601755s) [noise=true , address=77.90.31.14:4663 , piece_id=ISXY5DWAFL6JERL3OLJF6ECHONXJYIYJFB7FCIOJXX4EV6KF2C3Q, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:32.452746949 (25h19m44.928685414s) [noise=true , address=77.90.31.17:4772 , piece_id=TPUSDMZCAEDA7K5SMKR3NM7F3SDFXRXYTPYYAXAKZMF4I4APDHHA, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:32.502663977 (25h19m44.978602442s) [noise=true , address=104.36.82.8:4969 , piece_id=6RSYLRP2ME6OKBDANNBD77TZEHWABCXCI6LVXD6KDMF46AAOWQLQ, seg_pos={0 0}, upload=24242]: Done uploading piece. err: <nil> Jul 29 18:05:32.502693494 (25h19m44.978631959s) [seg_pos={0 0}, upload=24242]: Segment reached optimal threshold of 13 pieces. Jul 29 18:05:32.502820617 (25h19m44.978759082s) [noise=true , address=104.36.82.9:4007 , piece_id=3ZBOLLCN6A7YQ2UIY3OU5NUGOUGVSBANC6SY2MYIK3VADF3QOLVQ, seg_pos={0 0}, upload=24242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Jul 29 18:05:32.502839684 (25h19m44.978778149s) [noise=true , address=77.90.31.11:4553 , piece_id=FVJHS7V5GKQUR4DEJXKEMHBPJM6MYAWFPDC7JCQXJEE5KFLXJC3Q, seg_pos={0 0}, upload=24242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Jul 29 18:05:32.502895421 (25h19m44.978833896s) [noise=true , address=77.90.31.18:4806 , piece_id=SNX5YHNAOBZWR32HKHWCV4L3HDK6HOTF7SOU7WG7DSOHTZC77D7A, seg_pos={0 0}, upload=24242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Jul 29 18:05:32.502914306 (25h19m44.978852781s) [seg_pos={0 0}, upload=24242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:05:32.503010881 (25h19m44.978949346s) [upload=24242]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:05:32.503016191 (25h19m44.978954656s) [upload=24242]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:05:32.972533130 (25h19m45.448471595s) [upload=24242]: Done uploading segments... Jul 29 18:05:32.972551957 (25h19m45.448490412s) [upload=24242]: Upload finished. err: <nil> Duration: 30.001943685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X...X.................................X..................] duration: 1.92425508s FAIL (2025-07-29 18:05:55 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.92430171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [................................X..........X.XX......XX.XX.X] duration: 30.002246295s FAIL (2025-07-29 18:05:26 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002267075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.00244689s FAIL (2025-07-29 18:05:07 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4876: use of closed network connection; write tcp 188.166.225.170:50358->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50358->104.36.82.6:4876: use of closed network connection Jul 29 18:03:36.280903070 (25h17m48.756841535s) [seg_pos={0 0}, upload=24215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:03:36.280964928 (25h17m48.756903393s) [upload=24215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:03:36.280968505 (25h17m48.75690697s) [upload=24215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:03:36.580401990 (25h17m49.056340455s) [upload=24215]: Done uploading segments... Jul 29 18:03:36.580420275 (25h17m49.05635874s) [upload=24215]: Upload finished. err: <nil> Jul 29 18:04:37.738415110 (25h18m50.214353565s) [upload=24231]: Starting upload Jul 29 18:04:37.738467721 (25h18m50.214406176s) [upload=24231]: Done starting upload Jul 29 18:04:37.738521855 (25h18m50.21446032s) [upload=24231]: Uploading segments... Jul 29 18:04:37.738659808 (25h18m50.214598274s) [upload=24231]: Got next segment. Inline: false Jul 29 18:04:37.738684746 (25h18m50.214623211s) [upload=24231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 18:04:37.738688914 (25h18m50.214627379s) [upload=24231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 18:04:37.974714369 (25h18m50.450652834s) [seg_pos={0 0}, upload=24231]: Begin upload segment... Jul 29 18:04:37.974773131 (25h18m50.450711596s) [seg_pos={0 0}, upload=24231]: Done begin upload segment. Jul 29 18:04:37.974794843 (25h18m50.450733308s) [noise=true , address=45.91.116.10:4083 , piece_id=NO77ZLIGYMBJOW4XS2EW2VVWAJJBCZQV7JT66QGNPF4PVUMPAJ7A, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.974838666 (25h18m50.450777131s) [noise=true , address=104.36.82.16:4295 , piece_id=CO4OE575M6EQXZP2KBPFLP57SQJMKGWKVT6E2HSS4GVACW75BECA, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.975210127 (25h18m50.451148592s) [noise=true , address=45.91.116.11:4118 , piece_id=GCJPBH7YYLU5V4KIMLMJ6GH67D5D43OIKRQX3YGMQWAKIAYHKNHQ, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.975325448 (25h18m50.451263913s) [noise=true , address=45.91.116.13:4154 , piece_id=YTYBO3FGNXYRCZLSNAKTUUT6BQDHWKLXFEN34OD7H7U6W4S4XO6A, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.975471667 (25h18m50.451410122s) [noise=true , address=45.91.116.5:4299 , piece_id=QU6ETEVLPAQY5YDF33435RD54MB6E27D2KWEMDARPPU5PWQMLT7Q, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.975630882 (25h18m50.451569347s) [noise=true , address=77.90.31.14:4663 , piece_id=RF6KXZ6KEX7SCTZC3DGRBXYVYGEL7PUXVOCWZTMKGSDJEL5UYFZA, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.975859649 (25h18m50.451798114s) [noise=true , address=104.36.82.4:4801 , piece_id=ORFMROERL57YQNCEIXTRWICOPNRTHVVZEOTCRTQI23VU6ECXH2RQ, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.976123605 (25h18m50.45206207s) [noise=true , address=104.36.82.7:4910 , piece_id=7YGVSFGWNC63F7MC6VXVCYXQDIGYNHSHMCVCVVCTJ2DFTYE2KWVA, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.976382981 (25h18m50.452321446s) [noise=true , address=45.91.116.14:4190 , piece_id=J26BTFI5U7AXWC4X2M2F2JHXN3QQD2C32QCU3S34HEJTOUNCF4UQ, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.976675641 (25h18m50.452614096s) [noise=true , address=45.91.116.6:4974 , piece_id=MIZHS2QYMIATZAD3H262GWSNXXAMP7V42VQEGU7SV7VJPPM7WQBQ, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.976878219 (25h18m50.452816684s) [noise=true , address=77.90.31.15:4699 , piece_id=5FQD72IB47PHRGSTYHFCDM53IK2KSP4VD322TAZA72YD636QVSLQ, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.977069153 (25h18m50.453007618s) [noise=true , address=104.36.82.4:4802 , piece_id=N7JMKKJFLXVINPDIOHEEBF3IY3MQ5JCYNFXQNI5CUGN7MXT5PALQ, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.977279286 (25h18m50.453217751s) [noise=true , address=77.90.31.10:4517 , piece_id=PBWUXJFF2KSYOIMVXKO2W3VREW3M6SIP6TBXZGGEKQTKBLAHVXNA, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.978133701 (25h18m50.454072176s) [noise=true , address=77.90.31.12:4590 , piece_id=V5GNOM2XCXMFOGC2LFVMO5BGACSW37BOEOBKHPQLY4TKSOCXYMVA, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.978578932 (25h18m50.454517397s) [noise=true , address=77.90.31.21:4878 , piece_id=35WEE5PRGXY4D3ZT5IFEXSS4UVAACRLEVTEXVIQNLBD3ZILOHPMA, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:37.978877754 (25h18m50.454816219s) [noise=true , address=77.90.31.19:4841 , piece_id=3KNKJM323L4KDLP35NKIS7HZGGU4YIOIHV54WMPOQY5IREVLLW6A, seg_pos={0 0}, upload=24231]: Uploading piece... Jul 29 18:04:40.031482569 (25h18m52.507421034s) [upload=24231]: Next returned nil segment Jul 29 18:04:40.031497318 (25h18m52.507435783s) [upload=24231]: Waiting for error group managing segments... Jul 29 18:04:40.366807625 (25h18m52.84274609s) [noise=true , address=104.36.82.4:4802 , piece_id=N7JMKKJFLXVINPDIOHEEBF3IY3MQ5JCYNFXQNI5CUGN7MXT5PALQ, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.394982342 (25h18m52.870920827s) [noise=true , address=77.90.31.14:4663 , piece_id=RF6KXZ6KEX7SCTZC3DGRBXYVYGEL7PUXVOCWZTMKGSDJEL5UYFZA, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.418234261 (25h18m52.894172726s) [noise=true , address=45.91.116.10:4083 , piece_id=NO77ZLIGYMBJOW4XS2EW2VVWAJJBCZQV7JT66QGNPF4PVUMPAJ7A, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.495949506 (25h18m52.971887971s) [noise=true , address=104.36.82.4:4801 , piece_id=ORFMROERL57YQNCEIXTRWICOPNRTHVVZEOTCRTQI23VU6ECXH2RQ, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.502738256 (25h18m52.978676721s) [noise=true , address=77.90.31.21:4878 , piece_id=35WEE5PRGXY4D3ZT5IFEXSS4UVAACRLEVTEXVIQNLBD3ZILOHPMA, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.516995765 (25h18m52.99293423s) [noise=true , address=45.91.116.5:4299 , piece_id=QU6ETEVLPAQY5YDF33435RD54MB6E27D2KWEMDARPPU5PWQMLT7Q, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.518122691 (25h18m52.994061156s) [noise=true , address=45.91.116.6:4974 , piece_id=MIZHS2QYMIATZAD3H262GWSNXXAMP7V42VQEGU7SV7VJPPM7WQBQ, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.544864777 (25h18m53.020803242s) [noise=true , address=77.90.31.15:4699 , piece_id=5FQD72IB47PHRGSTYHFCDM53IK2KSP4VD322TAZA72YD636QVSLQ, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.578072334 (25h18m53.054010799s) [noise=true , address=77.90.31.19:4841 , piece_id=3KNKJM323L4KDLP35NKIS7HZGGU4YIOIHV54WMPOQY5IREVLLW6A, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.608525722 (25h18m53.084464187s) [noise=true , address=45.91.116.14:4190 , piece_id=J26BTFI5U7AXWC4X2M2F2JHXN3QQD2C32QCU3S34HEJTOUNCF4UQ, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.647568904 (25h18m53.123507369s) [noise=true , address=104.36.82.16:4295 , piece_id=CO4OE575M6EQXZP2KBPFLP57SQJMKGWKVT6E2HSS4GVACW75BECA, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:40.884037216 (25h18m53.359975681s) [noise=true , address=104.36.82.7:4910 , piece_id=7YGVSFGWNC63F7MC6VXVCYXQDIGYNHSHMCVCVVCTJ2DFTYE2KWVA, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:41.042983211 (25h18m53.518921676s) [noise=true , address=45.91.116.13:4154 , piece_id=YTYBO3FGNXYRCZLSNAKTUUT6BQDHWKLXFEN34OD7H7U6W4S4XO6A, seg_pos={0 0}, upload=24231]: Done uploading piece. err: <nil> Jul 29 18:04:41.043003030 (25h18m53.518941495s) [seg_pos={0 0}, upload=24231]: Segment reached optimal threshold of 13 pieces. Jul 29 18:04:41.043142627 (25h18m53.519081092s) [noise=true , address=77.90.31.10:4517 , piece_id=PBWUXJFF2KSYOIMVXKO2W3VREW3M6SIP6TBXZGGEKQTKBLAHVXNA, seg_pos={0 0}, upload=24231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Jul 29 18:04:41.043333953 (25h18m53.519272418s) [noise=true , address=77.90.31.12:4590 , piece_id=V5GNOM2XCXMFOGC2LFVMO5BGACSW37BOEOBKHPQLY4TKSOCXYMVA, seg_pos={0 0}, upload=24231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Jul 29 18:04:41.043381172 (25h18m53.519319627s) [noise=true , address=45.91.116.11:4118 , piece_id=GCJPBH7YYLU5V4KIMLMJ6GH67D5D43OIKRQX3YGMQWAKIAYHKNHQ, seg_pos={0 0}, upload=24231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:56648->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:56648->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56648->45.91.116.11:4118: use of closed network connection Jul 29 18:04:41.043389960 (25h18m53.519328425s) [seg_pos={0 0}, upload=24231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 29 18:04:41.043545126 (25h18m53.519483591s) [upload=24231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 29 18:04:41.043550778 (25h18m53.519489243s) [upload=24231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 29 18:04:41.465788927 (25h18m53.941727392s) [upload=24231]: Done uploading segments... Jul 29 18:04:41.465842159 (25h18m53.941780624s) [upload=24231]: Upload finished. err: <nil> Duration: 30.002470283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X...X.................................X.................] duration: 1.900530248s FAIL (2025-07-29 18:04:56 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.900554635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................X..........X.XX......XX.XX.] duration: 23.650023131s FAIL (2025-07-29 18:04:05 UTC) orbiter warning: uplink: context deadline exceeded Duration: 23.650041428s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)