Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 493294 reports (Page 5693 of 9866)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.645231482s FAIL (2025-10-06 23:16:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.645288852s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................XX........X...................X.] duration: 6.128671757s FAIL (2025-10-06 23:16:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.128701814s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X...........XX.......X..X..X...X.] duration: 7.380839764s FAIL (2025-10-06 23:16:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 6 23:15:09.154211438 (321h43m54.616303921s) [upload=283150]: Starting upload Oct 6 23:15:09.154301961 (321h43m54.616394434s) [upload=283150]: Done starting upload Oct 6 23:15:09.154464930 (321h43m54.616557404s) [upload=283150]: Uploading segments... Oct 6 23:15:09.154575902 (321h43m54.616668385s) [upload=283150]: Got next segment. Inline: false Oct 6 23:15:09.154635816 (321h43m54.616728309s) [upload=283150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:15:09.154643070 (321h43m54.616735553s) [upload=283150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:15:09.404324136 (321h43m54.866416619s) [seg_pos={0 0}, upload=283150]: Begin upload segment... Oct 6 23:15:09.404414268 (321h43m54.866506751s) [seg_pos={0 0}, upload=283150]: Done begin upload segment. Oct 6 23:15:09.404457700 (321h43m54.866550173s) [noise=true , address=77.90.31.18:4807 , piece_id=2VPKM6UHZ7L6AF3M7JQA6622GOCURMMGTVLF2MPFXTXLD375FCPQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.404493709 (321h43m54.866586192s) [noise=true , address=45.91.116.4:4940 , piece_id=HGOZ2BQEO4DRLV7FTBT3SG3BB7S2VOBUJVBI4PUCYB7Q2VEDGVJA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.405351473 (321h43m54.867443956s) [noise=true , address=145.79.227.4:4185 , piece_id=IYF64GCU7M46JMWQIV7QCCX3KQEIUS4LLDG3NK4KR7XZ7KSLASDQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.405549490 (321h43m54.867641993s) [noise=true , address=145.79.227.14:4840 , piece_id=2FF6JEUS2KJFHJIUONFTKZTXCWHPCNQRJQI7TTB2R4M6TGSWJMTA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.406049923 (321h43m54.868142396s) [noise=true , address=45.91.116.6:4974 , piece_id=QOIJAJ7CTQUZINXWQH2QB33NIBVCOI5JPZG4ELZF36FAUTK7UYRQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.406462478 (321h43m54.868554961s) [noise=true , address=45.91.116.14:4191 , piece_id=HYYUALQ3ZGV7U6677WN5PD6WUV5N5KOII3ZGPWSFYQ2X2BAOTWPQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.406851819 (321h43m54.868944292s) [noise=true , address=145.79.227.4:4187 , piece_id=WQPJGHBZYZN7AZXGVMODPUNL7TQTRWSZJUKWIK3PG66F7CVSK35A, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.407162331 (321h43m54.869254804s) [noise=true , address=145.79.227.3:4151 , piece_id=2SOR7RAPTC4R67ZXBNVR5ZGHGB2VLFRFK2LJN2EW2OHQ2L32Z3SQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.407241121 (321h43m54.869333594s) [noise=true , address=145.79.227.18:4006 , piece_id=PUIVI35AF63USYETIAE4A7TZRLLOKFKSQAD3TG7GT7GFYITYOVEA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.407573083 (321h43m54.869665556s) [noise=true , address=145.79.227.15:4873 , piece_id=ED7N4VPCONERQHIFYJ2H3MWGZYGMMRPGUGHBCX5GYIWQLWHZ7HNQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.407635571 (321h43m54.869728054s) [noise=true , address=145.79.227.7:4293 , piece_id=HOFGZ36EITH7EGIVT2LJRWRVAY7IEPJK3UJDRKJXNLGSNAWXRWPA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.408013451 (321h43m54.870105924s) [noise=true , address=145.79.227.16:4912 , piece_id=HAEY263FPZ7XCBSS76PPUPWWQGQO5DG2NREKQC2EKFIOU2VT3DEA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.408416769 (321h43m54.870509273s) [noise=true , address=45.91.116.5:4298 , piece_id=RYZO2XJCXK4DFG2JYAIT7WRF5UUYYBIC3RXDERJVBLHGTWZOAQEQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.408869872 (321h43m54.870962385s) [noise=true , address=77.90.31.10:4520 , piece_id=XGKZKGYMSXU45LLMWEANLPSIM7UE4F7IJYNJQ3Z3JASMECVSXVXA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.408975954 (321h43m54.871068437s) [noise=true , address=145.79.227.6:4257 , piece_id=WAH6JO4MLGPNVQSLTP74WZI53HPWRUL6OAYALKWN2CXKZUYV7ORA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.409374022 (321h43m54.871466505s) [noise=true , address=145.79.227.17:4969 , piece_id=K442RKRJNWE2FNUJGJX3QVUMY3Q6XVXC4ITJ3XAJBSK24KGMXG2A, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:11.091861874 (321h43m56.553954357s) [upload=283150]: Next returned nil segment Oct 6 23:15:11.091885419 (321h43m56.553977902s) [upload=283150]: Waiting for error group managing segments... Oct 6 23:15:11.371581328 (321h43m56.833673801s) [noise=true , address=145.79.227.4:4185 , piece_id=IYF64GCU7M46JMWQIV7QCCX3KQEIUS4LLDG3NK4KR7XZ7KSLASDQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.371695697 (321h43m56.83378817s) [noise=true , address=145.79.227.3:4151 , piece_id=2SOR7RAPTC4R67ZXBNVR5ZGHGB2VLFRFK2LJN2EW2OHQ2L32Z3SQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.386575500 (321h43m56.848667983s) [noise=true , address=145.79.227.18:4006 , piece_id=PUIVI35AF63USYETIAE4A7TZRLLOKFKSQAD3TG7GT7GFYITYOVEA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.387963022 (321h43m56.850055505s) [noise=true , address=145.79.227.7:4293 , piece_id=HOFGZ36EITH7EGIVT2LJRWRVAY7IEPJK3UJDRKJXNLGSNAWXRWPA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.388405806 (321h43m56.850498289s) [noise=true , address=145.79.227.14:4840 , piece_id=2FF6JEUS2KJFHJIUONFTKZTXCWHPCNQRJQI7TTB2R4M6TGSWJMTA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.392145927 (321h43m56.8542384s) [noise=true , address=145.79.227.4:4187 , piece_id=WQPJGHBZYZN7AZXGVMODPUNL7TQTRWSZJUKWIK3PG66F7CVSK35A, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.405761634 (321h43m56.867854107s) [noise=true , address=145.79.227.16:4912 , piece_id=HAEY263FPZ7XCBSS76PPUPWWQGQO5DG2NREKQC2EKFIOU2VT3DEA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.407616405 (321h43m56.869708878s) [noise=true , address=145.79.227.17:4969 , piece_id=K442RKRJNWE2FNUJGJX3QVUMY3Q6XVXC4ITJ3XAJBSK24KGMXG2A, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.408420748 (321h43m56.870513231s) [noise=true , address=145.79.227.6:4257 , piece_id=WAH6JO4MLGPNVQSLTP74WZI53HPWRUL6OAYALKWN2CXKZUYV7ORA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.418159201 (321h43m56.880251684s) [noise=true , address=145.79.227.15:4873 , piece_id=ED7N4VPCONERQHIFYJ2H3MWGZYGMMRPGUGHBCX5GYIWQLWHZ7HNQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:35.486594858 (321h44m20.948687331s) [noise=true , address=45.91.116.6:4974 , piece_id=QOIJAJ7CTQUZINXWQH2QB33NIBVCOI5JPZG4ELZF36FAUTK7UYRQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:38.447472751 (321h44m23.909565234s) [noise=true , address=77.90.31.10:4520 , piece_id=XGKZKGYMSXU45LLMWEANLPSIM7UE4F7IJYNJQ3Z3JASMECVSXVXA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.447685205 (321h44m23.909777688s) [noise=true , address=45.91.116.14:4191 , piece_id=HYYUALQ3ZGV7U6677WN5PD6WUV5N5KOII3ZGPWSFYQ2X2BAOTWPQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: write tcp 188.166.225.170:47242->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.447890236 (321h44m23.909982719s) [noise=true , address=45.91.116.5:4298 , piece_id=RYZO2XJCXK4DFG2JYAIT7WRF5UUYYBIC3RXDERJVBLHGTWZOAQEQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.448073304 (321h44m23.910165777s) [noise=true , address=77.90.31.18:4807 , piece_id=2VPKM6UHZ7L6AF3M7JQA6622GOCURMMGTVLF2MPFXTXLD375FCPQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.448272945 (321h44m23.910365428s) [noise=true , address=45.91.116.4:4940 , piece_id=HGOZ2BQEO4DRLV7FTBT3SG3BB7S2VOBUJVBI4PUCYB7Q2VEDGVJA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 188.166.225.170:55146->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.448306679 (321h44m23.910399152s) [seg_pos={0 0}, upload=283150]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 6 23:15:38.699493642 (321h44m24.161586125s) [upload=283150]: Done uploading segments... Oct 6 23:15:38.699530192 (321h44m24.161622665s) [upload=283150]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.380872687s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.710049187s FAIL (2025-10-06 23:15:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.710078733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X..................X......X..X.] duration: 6.56257198s FAIL (2025-10-06 23:15:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment... Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment. Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments... Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments... Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 6.562592418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X..................X......X..X] duration: 1m0.252771875s FAIL (2025-10-06 23:15:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment... Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment. Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments... Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments... Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.252800199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X...........XX.......X..X..X...X] duration: 30.253897974s FAIL (2025-10-06 23:15:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 6 23:15:09.154211438 (321h43m54.616303921s) [upload=283150]: Starting upload Oct 6 23:15:09.154301961 (321h43m54.616394434s) [upload=283150]: Done starting upload Oct 6 23:15:09.154464930 (321h43m54.616557404s) [upload=283150]: Uploading segments... Oct 6 23:15:09.154575902 (321h43m54.616668385s) [upload=283150]: Got next segment. Inline: false Oct 6 23:15:09.154635816 (321h43m54.616728309s) [upload=283150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:15:09.154643070 (321h43m54.616735553s) [upload=283150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:15:09.404324136 (321h43m54.866416619s) [seg_pos={0 0}, upload=283150]: Begin upload segment... Oct 6 23:15:09.404414268 (321h43m54.866506751s) [seg_pos={0 0}, upload=283150]: Done begin upload segment. Oct 6 23:15:09.404457700 (321h43m54.866550173s) [noise=true , address=77.90.31.18:4807 , piece_id=2VPKM6UHZ7L6AF3M7JQA6622GOCURMMGTVLF2MPFXTXLD375FCPQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.404493709 (321h43m54.866586192s) [noise=true , address=45.91.116.4:4940 , piece_id=HGOZ2BQEO4DRLV7FTBT3SG3BB7S2VOBUJVBI4PUCYB7Q2VEDGVJA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.405351473 (321h43m54.867443956s) [noise=true , address=145.79.227.4:4185 , piece_id=IYF64GCU7M46JMWQIV7QCCX3KQEIUS4LLDG3NK4KR7XZ7KSLASDQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.405549490 (321h43m54.867641993s) [noise=true , address=145.79.227.14:4840 , piece_id=2FF6JEUS2KJFHJIUONFTKZTXCWHPCNQRJQI7TTB2R4M6TGSWJMTA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.406049923 (321h43m54.868142396s) [noise=true , address=45.91.116.6:4974 , piece_id=QOIJAJ7CTQUZINXWQH2QB33NIBVCOI5JPZG4ELZF36FAUTK7UYRQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.406462478 (321h43m54.868554961s) [noise=true , address=45.91.116.14:4191 , piece_id=HYYUALQ3ZGV7U6677WN5PD6WUV5N5KOII3ZGPWSFYQ2X2BAOTWPQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.406851819 (321h43m54.868944292s) [noise=true , address=145.79.227.4:4187 , piece_id=WQPJGHBZYZN7AZXGVMODPUNL7TQTRWSZJUKWIK3PG66F7CVSK35A, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.407162331 (321h43m54.869254804s) [noise=true , address=145.79.227.3:4151 , piece_id=2SOR7RAPTC4R67ZXBNVR5ZGHGB2VLFRFK2LJN2EW2OHQ2L32Z3SQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.407241121 (321h43m54.869333594s) [noise=true , address=145.79.227.18:4006 , piece_id=PUIVI35AF63USYETIAE4A7TZRLLOKFKSQAD3TG7GT7GFYITYOVEA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.407573083 (321h43m54.869665556s) [noise=true , address=145.79.227.15:4873 , piece_id=ED7N4VPCONERQHIFYJ2H3MWGZYGMMRPGUGHBCX5GYIWQLWHZ7HNQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.407635571 (321h43m54.869728054s) [noise=true , address=145.79.227.7:4293 , piece_id=HOFGZ36EITH7EGIVT2LJRWRVAY7IEPJK3UJDRKJXNLGSNAWXRWPA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.408013451 (321h43m54.870105924s) [noise=true , address=145.79.227.16:4912 , piece_id=HAEY263FPZ7XCBSS76PPUPWWQGQO5DG2NREKQC2EKFIOU2VT3DEA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.408416769 (321h43m54.870509273s) [noise=true , address=45.91.116.5:4298 , piece_id=RYZO2XJCXK4DFG2JYAIT7WRF5UUYYBIC3RXDERJVBLHGTWZOAQEQ, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.408869872 (321h43m54.870962385s) [noise=true , address=77.90.31.10:4520 , piece_id=XGKZKGYMSXU45LLMWEANLPSIM7UE4F7IJYNJQ3Z3JASMECVSXVXA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.408975954 (321h43m54.871068437s) [noise=true , address=145.79.227.6:4257 , piece_id=WAH6JO4MLGPNVQSLTP74WZI53HPWRUL6OAYALKWN2CXKZUYV7ORA, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:09.409374022 (321h43m54.871466505s) [noise=true , address=145.79.227.17:4969 , piece_id=K442RKRJNWE2FNUJGJX3QVUMY3Q6XVXC4ITJ3XAJBSK24KGMXG2A, seg_pos={0 0}, upload=283150]: Uploading piece... Oct 6 23:15:11.091861874 (321h43m56.553954357s) [upload=283150]: Next returned nil segment Oct 6 23:15:11.091885419 (321h43m56.553977902s) [upload=283150]: Waiting for error group managing segments... Oct 6 23:15:11.371581328 (321h43m56.833673801s) [noise=true , address=145.79.227.4:4185 , piece_id=IYF64GCU7M46JMWQIV7QCCX3KQEIUS4LLDG3NK4KR7XZ7KSLASDQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.371695697 (321h43m56.83378817s) [noise=true , address=145.79.227.3:4151 , piece_id=2SOR7RAPTC4R67ZXBNVR5ZGHGB2VLFRFK2LJN2EW2OHQ2L32Z3SQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.386575500 (321h43m56.848667983s) [noise=true , address=145.79.227.18:4006 , piece_id=PUIVI35AF63USYETIAE4A7TZRLLOKFKSQAD3TG7GT7GFYITYOVEA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.387963022 (321h43m56.850055505s) [noise=true , address=145.79.227.7:4293 , piece_id=HOFGZ36EITH7EGIVT2LJRWRVAY7IEPJK3UJDRKJXNLGSNAWXRWPA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.388405806 (321h43m56.850498289s) [noise=true , address=145.79.227.14:4840 , piece_id=2FF6JEUS2KJFHJIUONFTKZTXCWHPCNQRJQI7TTB2R4M6TGSWJMTA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.392145927 (321h43m56.8542384s) [noise=true , address=145.79.227.4:4187 , piece_id=WQPJGHBZYZN7AZXGVMODPUNL7TQTRWSZJUKWIK3PG66F7CVSK35A, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.405761634 (321h43m56.867854107s) [noise=true , address=145.79.227.16:4912 , piece_id=HAEY263FPZ7XCBSS76PPUPWWQGQO5DG2NREKQC2EKFIOU2VT3DEA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.407616405 (321h43m56.869708878s) [noise=true , address=145.79.227.17:4969 , piece_id=K442RKRJNWE2FNUJGJX3QVUMY3Q6XVXC4ITJ3XAJBSK24KGMXG2A, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.408420748 (321h43m56.870513231s) [noise=true , address=145.79.227.6:4257 , piece_id=WAH6JO4MLGPNVQSLTP74WZI53HPWRUL6OAYALKWN2CXKZUYV7ORA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:11.418159201 (321h43m56.880251684s) [noise=true , address=145.79.227.15:4873 , piece_id=ED7N4VPCONERQHIFYJ2H3MWGZYGMMRPGUGHBCX5GYIWQLWHZ7HNQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:35.486594858 (321h44m20.948687331s) [noise=true , address=45.91.116.6:4974 , piece_id=QOIJAJ7CTQUZINXWQH2QB33NIBVCOI5JPZG4ELZF36FAUTK7UYRQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: <nil> Oct 6 23:15:38.447472751 (321h44m23.909565234s) [noise=true , address=77.90.31.10:4520 , piece_id=XGKZKGYMSXU45LLMWEANLPSIM7UE4F7IJYNJQ3Z3JASMECVSXVXA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.447685205 (321h44m23.909777688s) [noise=true , address=45.91.116.14:4191 , piece_id=HYYUALQ3ZGV7U6677WN5PD6WUV5N5KOII3ZGPWSFYQ2X2BAOTWPQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: write tcp 188.166.225.170:47242->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.447890236 (321h44m23.909982719s) [noise=true , address=45.91.116.5:4298 , piece_id=RYZO2XJCXK4DFG2JYAIT7WRF5UUYYBIC3RXDERJVBLHGTWZOAQEQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.448073304 (321h44m23.910165777s) [noise=true , address=77.90.31.18:4807 , piece_id=2VPKM6UHZ7L6AF3M7JQA6622GOCURMMGTVLF2MPFXTXLD375FCPQ, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.448272945 (321h44m23.910365428s) [noise=true , address=45.91.116.4:4940 , piece_id=HGOZ2BQEO4DRLV7FTBT3SG3BB7S2VOBUJVBI4PUCYB7Q2VEDGVJA, seg_pos={0 0}, upload=283150]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 188.166.225.170:55146->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:38.448306679 (321h44m23.910399152s) [seg_pos={0 0}, upload=283150]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 6 23:15:38.699493642 (321h44m24.161586125s) [upload=283150]: Done uploading segments... Oct 6 23:15:38.699530192 (321h44m24.161622665s) [upload=283150]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.253928102s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX........X...................X] duration: 30.001439107s FAIL (2025-10-06 23:15:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001508048s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.664058626s FAIL (2025-10-06 23:15:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.664088804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX........X...................] duration: 6.125697983s FAIL (2025-10-06 23:14:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.125728472s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.703355881s FAIL (2025-10-06 23:14:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.703387481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X...........XX.......X..X..X...] duration: 6.573666194s FAIL (2025-10-06 23:14:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ) [upload=283103]: Done starting upload Oct 6 23:11:39.165713278 (321h40m24.627805761s) [upload=283103]: Uploading segments... Oct 6 23:11:39.165926464 (321h40m24.628018957s) [upload=283103]: Got next segment. Inline: false Oct 6 23:11:39.166463165 (321h40m24.628555638s) [upload=283103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:11:39.166468907 (321h40m24.62856139s) [upload=283103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:11:39.416464112 (321h40m24.878556585s) [seg_pos={0 0}, upload=283103]: Begin upload segment... Oct 6 23:11:39.416533804 (321h40m24.878626277s) [seg_pos={0 0}, upload=283103]: Done begin upload segment. Oct 6 23:11:39.416562188 (321h40m24.878654661s) [noise=true , address=145.79.227.4:4185 , piece_id=G75L47ZLAM73SH4UW5HYVSJKHWUN7WYCMGH3HNPSZ67PASTAYVSA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.416623334 (321h40m24.878715817s) [noise=true , address=145.79.227.13:4802 , piece_id=SLDTYBRLPKJBWI6TH6NBW34EV3R537UEUJJZFGFYKUKKMHUYPECA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.417313729 (321h40m24.879406212s) [noise=true , address=77.90.31.13:4627 , piece_id=KRXAAGW63PKK5DY5MJZF5UPQLIPX52CKKWA3ZFH26WBTZCBE6ORQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.417781509 (321h40m24.879874002s) [noise=true , address=145.79.227.2:4115 , piece_id=CABLKD7HYRPA74BXVXKLEBTDFWAOPY66H76XYY4BAR5QOJ46DJQQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.418910009 (321h40m24.881002482s) [noise=true , address=145.79.227.16:4911 , piece_id=KRVX3BN2PSBZSKNW3XFPKBREEJYCFWD2PJCIP7FHF6JJWYPGSNKQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.419368221 (321h40m24.881460704s) [noise=true , address=145.79.227.6:4257 , piece_id=L5AF3ISZBP6QF3TZFBQW2MS5KM27Y4JBBONSLHO6CHJZEGQZQI4Q, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.420810096 (321h40m24.882902569s) [noise=true , address=77.90.31.11:4553 , piece_id=IOOB6KDBP32D4WIHQ3JCWBFUO7ITOIJMYQBQEQ6QZV2CWURHM6IA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.421327622 (321h40m24.883420105s) [noise=true , address=77.90.31.15:4697 , piece_id=FYHG55O4FUO6DLGJEU6MCHB3ORBRL5G6JGV2YGO3RGUDHUXE4AQA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.422674918 (321h40m24.884767401s) [noise=true , address=145.79.227.19:4077 , piece_id=NMQQC6MHTFNASAQ4WOODCRPYCVNM4FQYVBILEKNB7EHP4IPRTYCA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.423322712 (321h40m24.885415185s) [noise=true , address=77.90.31.16:4734 , piece_id=5QS6YKX3ZRX6KQWTMUGMQCUTK3ON4LCUYZUS5B2DE2UQUTHGSRUA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.425171101 (321h40m24.887263584s) [noise=true , address=77.90.31.21:4877 , piece_id=63R63JLRLK7UGVNCUOIXTFHKW3OKFHXFGUXD5D6C2RJC4MVBMJZQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.426722144 (321h40m24.888814627s) [noise=true , address=145.79.227.7:4294 , piece_id=MH7WZ6AWGTDRTHIZKSTZVR34JAZTM3HO2DGKMOXRATLLOPBVSLVQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.428244814 (321h40m24.890337297s) [noise=true , address=45.91.116.11:4118 , piece_id=KR4UNHZZZXWRI4OEPJ4ALYAPCMMWOWDHSF6DQK7FKQMZQ5WBY5IQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.429659388 (321h40m24.891751861s) [noise=true , address=77.90.31.14:4661 , piece_id=A4YLZTZCGXDYQCXSXBFIGLEXHQYNNXH6LMSA7RE6LNUTECPELKMQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.430972938 (321h40m24.893065411s) [noise=true , address=145.79.227.19:4079 , piece_id=CYJWBNPZA3WY3GF5NKFUYR2NI734LUXUP4MS2FWHZRBAE233V2PA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.431406765 (321h40m24.893499238s) [noise=true , address=77.90.31.19:4841 , piece_id=I4NNAD37P6MJX6HELBMJVOK4ZUUMIXTQUHR72TVFATV6ZUJZM2DQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:41.114941991 (321h40m26.577034464s) [upload=283103]: Next returned nil segment Oct 6 23:11:41.114961708 (321h40m26.577054191s) [upload=283103]: Waiting for error group managing segments... Oct 6 23:11:41.395880787 (321h40m26.857973271s) [noise=true , address=145.79.227.2:4115 , piece_id=CABLKD7HYRPA74BXVXKLEBTDFWAOPY66H76XYY4BAR5QOJ46DJQQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.396221086 (321h40m26.858313569s) [noise=true , address=145.79.227.19:4077 , piece_id=NMQQC6MHTFNASAQ4WOODCRPYCVNM4FQYVBILEKNB7EHP4IPRTYCA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.400339508 (321h40m26.862431981s) [noise=true , address=145.79.227.13:4802 , piece_id=SLDTYBRLPKJBWI6TH6NBW34EV3R537UEUJJZFGFYKUKKMHUYPECA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.404391554 (321h40m26.866484027s) [noise=true , address=145.79.227.6:4257 , piece_id=L5AF3ISZBP6QF3TZFBQW2MS5KM27Y4JBBONSLHO6CHJZEGQZQI4Q, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.410508050 (321h40m26.872600523s) [noise=true , address=145.79.227.7:4294 , piece_id=MH7WZ6AWGTDRTHIZKSTZVR34JAZTM3HO2DGKMOXRATLLOPBVSLVQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.422887814 (321h40m26.884980287s) [noise=true , address=145.79.227.19:4079 , piece_id=CYJWBNPZA3WY3GF5NKFUYR2NI734LUXUP4MS2FWHZRBAE233V2PA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.423758171 (321h40m26.885850644s) [noise=true , address=145.79.227.4:4185 , piece_id=G75L47ZLAM73SH4UW5HYVSJKHWUN7WYCMGH3HNPSZ67PASTAYVSA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.493892409 (321h40m26.955984892s) [noise=true , address=145.79.227.16:4911 , piece_id=KRVX3BN2PSBZSKNW3XFPKBREEJYCFWD2PJCIP7FHF6JJWYPGSNKQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:12:08.456557177 (321h40m53.91864966s) [noise=true , address=77.90.31.14:4661 , piece_id=A4YLZTZCGXDYQCXSXBFIGLEXHQYNNXH6LMSA7RE6LNUTECPELKMQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.456872789 (321h40m53.918965282s) [noise=true , address=77.90.31.13:4627 , piece_id=KRXAAGW63PKK5DY5MJZF5UPQLIPX52CKKWA3ZFH26WBTZCBE6ORQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: rpc: dial tcp 77.90.31.13:4627: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457082308 (321h40m53.919174781s) [noise=true , address=45.91.116.11:4118 , piece_id=KR4UNHZZZXWRI4OEPJ4ALYAPCMMWOWDHSF6DQK7FKQMZQ5WBY5IQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457260407 (321h40m53.91935288s) [noise=true , address=77.90.31.15:4697 , piece_id=FYHG55O4FUO6DLGJEU6MCHB3ORBRL5G6JGV2YGO3RGUDHUXE4AQA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457432174 (321h40m53.919524657s) [noise=true , address=77.90.31.16:4734 , piece_id=5QS6YKX3ZRX6KQWTMUGMQCUTK3ON4LCUYZUS5B2DE2UQUTHGSRUA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457616485 (321h40m53.919708968s) [noise=true , address=77.90.31.19:4841 , piece_id=I4NNAD37P6MJX6HELBMJVOK4ZUUMIXTQUHR72TVFATV6ZUJZM2DQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 188.166.225.170:45634->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457895887 (321h40m53.91998837s) [noise=true , address=77.90.31.11:4553 , piece_id=IOOB6KDBP32D4WIHQ3JCWBFUO7ITOIJMYQBQEQ6QZV2CWURHM6IA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 188.166.225.170:42640->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.458065830 (321h40m53.920158304s) [noise=true , address=77.90.31.21:4877 , piece_id=63R63JLRLK7UGVNCUOIXTFHKW3OKFHXFGUXD5D6C2RJC4MVBMJZQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:42068->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.458094555 (321h40m53.920187038s) [seg_pos={0 0}, upload=283103]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 6 23:12:08.698421029 (321h40m54.160513512s) [upload=283103]: Done uploading segments... Oct 6 23:12:08.698456296 (321h40m54.160548779s) [upload=283103]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.5737007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XX........X..................] duration: 6.338179569s FAIL (2025-10-06 23:13:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.338226369s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..................X......X..] duration: 6.477353206s FAIL (2025-10-06 23:13:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 7400762s) [noise=true , address=145.79.227.31:4820 , piece_id=KDF3S2SFAHERKYTZXLR4UPOOXPZ475ORJCO7S3BOJGMW3NKBY4GQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.412288768 (321h40m20.874381241s) [noise=true , address=77.90.31.18:4831 , piece_id=IKUBFGKWHIKFHVSPAKISHQ2A3FQOQTWD3YVNKXLOREBL4KPFZSYQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.412644696 (321h40m20.874737169s) [noise=true , address=64.49.13.137:4502 , piece_id=W7SGEHG7ZLLYMRM4Z4XZ3CTSZJJDM6BGUV6IXOC7VXEC7YPLUG2Q, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413017876 (321h40m20.875110359s) [noise=true , address=45.91.116.7:4027 , piece_id=UEA4UJLNVUTWSNL6NUUN7SLCHY2BZ7SC6NQAHNCSMSQX474OL2JQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413396136 (321h40m20.875488619s) [noise=true , address=77.90.31.19:4851 , piece_id=53XK564AIKRQUC5CIVJLDPPMCKLA46WNSTAPJX46QJHJQ6QMEAFQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413895047 (321h40m20.87598752s) [noise=true , address=45.91.116.11:4129 , piece_id=KIAPVZXQ4QSEEAHXA5JCBKNJOS5D77B336O6QEFAABYI2WCBEFQA, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.414559813 (321h40m20.876652296s) [noise=true , address=145.79.227.29:4943 , piece_id=LAOUJXCAYRCVWU3VDX76FPB5F3HOYZEA3YVDT4F5KE4UG2OMZB6Q, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415078200 (321h40m20.877170683s) [noise=true , address=77.90.31.19:4869 , piece_id=WSJXCO7K2TG6WYVT6GCSJBWNKCE2YRT4DYEGFMJZPVGZ7UDXDLYQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415439157 (321h40m20.87753163s) [noise=true , address=45.91.116.12:4241 , piece_id=CEDAYVITA66N2QA2KOCK7KIU3KBBYYS2VXWO4GPEWSVP7Z4RULNA, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415763504 (321h40m20.877855987s) [noise=true , address=45.91.116.13:4166 , piece_id=7QTSHYGDBUTFHUQDFNJYQYXN4V6IRU7IMQ6IFMKBFYVC3XDXICPQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.416122719 (321h40m20.878215202s) [noise=true , address=77.90.31.11:4579 , piece_id=NE4RUF4T6U7UOM4AMCVJNOOIQ5O5BHETJ3YZCGVG7EI44SQUSMFQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:36.702309353 (321h40m22.164401836s) [upload=283102]: Next returned nil segment Oct 6 23:11:36.702329182 (321h40m22.164421665s) [upload=283102]: Waiting for error group managing segments... Oct 6 23:11:37.029061628 (321h40m22.491154101s) [noise=true , address=64.49.13.137:4502 , piece_id=W7SGEHG7ZLLYMRM4Z4XZ3CTSZJJDM6BGUV6IXOC7VXEC7YPLUG2Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.252437915 (321h40m22.714530398s) [noise=true , address=145.79.227.34:4929 , piece_id=NHKA33B6AQLEIRTLYQACKESLPOSM7WOABQOQDEBXUQYKDNXMDUCQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.256976217 (321h40m22.7190687s) [noise=true , address=145.79.227.21:4479 , piece_id=27LBIQRR5KL5MQGWWAMXQ3NWAYGRJT6RIELLLPFURPI6MMBVGMBA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.257492219 (321h40m22.719584692s) [noise=true , address=145.79.227.31:4820 , piece_id=KDF3S2SFAHERKYTZXLR4UPOOXPZ475ORJCO7S3BOJGMW3NKBY4GQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.277387253 (321h40m22.739479726s) [noise=true , address=145.79.227.29:4943 , piece_id=LAOUJXCAYRCVWU3VDX76FPB5F3HOYZEA3YVDT4F5KE4UG2OMZB6Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:12:34.456871161 (321h41m19.918963654s) [noise=true , address=77.90.31.19:4869 , piece_id=WSJXCO7K2TG6WYVT6GCSJBWNKCE2YRT4DYEGFMJZPVGZ7UDXDLYQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: rpc: dial tcp 77.90.31.19:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.456947426 (321h41m19.919039909s) [noise=true , address=77.90.31.19:4851 , piece_id=53XK564AIKRQUC5CIVJLDPPMCKLA46WNSTAPJX46QJHJQ6QMEAFQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457178967 (321h41m19.91927145s) [noise=true , address=77.90.31.11:4579 , piece_id=NE4RUF4T6U7UOM4AMCVJNOOIQ5O5BHETJ3YZCGVG7EI44SQUSMFQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: read tcp 188.166.225.170:58086->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457193044 (321h41m19.919285527s) [noise=true , address=45.91.116.6:4988 , piece_id=A2P6YDUI3S57WSUK7QJST6LXXNHDDUYF542MS7LU6OQF723J2QIQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS, address:45.91.116.6:4988): protocol: rpc: dial tcp 45.91.116.6:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457329935 (321h41m19.919422418s) [noise=true , address=45.91.116.5:4309 , piece_id=DKBEWGUHKP6IIVHO7S7SM2O2UXPVXQHJT6Q7GXJJNOZXWRRJA7KA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1CHPuJdsjXtHCho9WdqpSvzRbEJqmAXchTkhmXgVqWQTTTCrV3, address:45.91.116.5:4309): protocol: rpc: dial tcp 45.91.116.5:4309: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457367937 (321h41m19.91946042s) [noise=true , address=45.91.116.14:4195 , piece_id=NZ3BQUPMWERJ3J45MXI6WZOCATOLVZ7VMNULANLYHQBDRS4E657A, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457501181 (321h41m19.919593654s) [noise=true , address=45.91.116.7:4027 , piece_id=UEA4UJLNVUTWSNL6NUUN7SLCHY2BZ7SC6NQAHNCSMSQX474OL2JQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq, address:45.91.116.7:4027): protocol: write tcp 188.166.225.170:56512->45.91.116.7:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457509076 (321h41m19.919601569s) [noise=true , address=45.91.116.12:4241 , piece_id=CEDAYVITA66N2QA2KOCK7KIU3KBBYYS2VXWO4GPEWSVP7Z4RULNA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:128wA5sH9sQDinHwwiTGL7WDktf6yD8EqQqy9mCxeZAKorYgsXD, address:45.91.116.12:4241): protocol: read tcp 188.166.225.170:37074->45.91.116.12:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457609828 (321h41m19.919702301s) [noise=true , address=45.91.116.11:4129 , piece_id=KIAPVZXQ4QSEEAHXA5JCBKNJOS5D77B336O6QEFAABYI2WCBEFQA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: read tcp 188.166.225.170:36136->45.91.116.11:4129: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457634866 (321h41m19.919727349s) [noise=true , address=77.90.31.13:4636 , piece_id=EVMVXR62UI42REJ6GHRYY5EANJARGC3LZYPBDFSOIDS6AO72KQWQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: read tcp 188.166.225.170:37498->77.90.31.13:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457770924 (321h41m19.919863397s) [noise=true , address=77.90.31.21:4907 , piece_id=F64NQLEYP2CYXNBH7YT3EXGR35OCH2RPXN7M2F46SSHGIDE5NW5Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: read tcp 188.166.225.170:37048->77.90.31.21:4907: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457900290 (321h41m19.919992773s) [noise=true , address=77.90.31.18:4831 , piece_id=IKUBFGKWHIKFHVSPAKISHQ2A3FQOQTWD3YVNKXLOREBL4KPFZSYQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: read tcp 188.166.225.170:59108->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457935207 (321h41m19.9200277s) [noise=true , address=45.91.116.13:4166 , piece_id=7QTSHYGDBUTFHUQDFNJYQYXN4V6IRU7IMQ6IFMKBFYVC3XDXICPQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: read tcp 188.166.225.170:34944->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457964232 (321h41m19.920056725s) [seg_pos={0 0}, upload=283102]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Oct 6 23:12:34.699771335 (321h41m20.161863829s) [upload=283102]: Done uploading segments... Oct 6 23:12:34.699827862 (321h41m20.161920356s) [upload=283102]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 6.477384686s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.135160343s FAIL (2025-10-06 23:13:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.135192805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................X...........XX.......X..X..X..] duration: 7.286574758s FAIL (2025-10-06 23:13:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ) [upload=283103]: Done starting upload Oct 6 23:11:39.165713278 (321h40m24.627805761s) [upload=283103]: Uploading segments... Oct 6 23:11:39.165926464 (321h40m24.628018957s) [upload=283103]: Got next segment. Inline: false Oct 6 23:11:39.166463165 (321h40m24.628555638s) [upload=283103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:11:39.166468907 (321h40m24.62856139s) [upload=283103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:11:39.416464112 (321h40m24.878556585s) [seg_pos={0 0}, upload=283103]: Begin upload segment... Oct 6 23:11:39.416533804 (321h40m24.878626277s) [seg_pos={0 0}, upload=283103]: Done begin upload segment. Oct 6 23:11:39.416562188 (321h40m24.878654661s) [noise=true , address=145.79.227.4:4185 , piece_id=G75L47ZLAM73SH4UW5HYVSJKHWUN7WYCMGH3HNPSZ67PASTAYVSA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.416623334 (321h40m24.878715817s) [noise=true , address=145.79.227.13:4802 , piece_id=SLDTYBRLPKJBWI6TH6NBW34EV3R537UEUJJZFGFYKUKKMHUYPECA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.417313729 (321h40m24.879406212s) [noise=true , address=77.90.31.13:4627 , piece_id=KRXAAGW63PKK5DY5MJZF5UPQLIPX52CKKWA3ZFH26WBTZCBE6ORQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.417781509 (321h40m24.879874002s) [noise=true , address=145.79.227.2:4115 , piece_id=CABLKD7HYRPA74BXVXKLEBTDFWAOPY66H76XYY4BAR5QOJ46DJQQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.418910009 (321h40m24.881002482s) [noise=true , address=145.79.227.16:4911 , piece_id=KRVX3BN2PSBZSKNW3XFPKBREEJYCFWD2PJCIP7FHF6JJWYPGSNKQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.419368221 (321h40m24.881460704s) [noise=true , address=145.79.227.6:4257 , piece_id=L5AF3ISZBP6QF3TZFBQW2MS5KM27Y4JBBONSLHO6CHJZEGQZQI4Q, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.420810096 (321h40m24.882902569s) [noise=true , address=77.90.31.11:4553 , piece_id=IOOB6KDBP32D4WIHQ3JCWBFUO7ITOIJMYQBQEQ6QZV2CWURHM6IA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.421327622 (321h40m24.883420105s) [noise=true , address=77.90.31.15:4697 , piece_id=FYHG55O4FUO6DLGJEU6MCHB3ORBRL5G6JGV2YGO3RGUDHUXE4AQA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.422674918 (321h40m24.884767401s) [noise=true , address=145.79.227.19:4077 , piece_id=NMQQC6MHTFNASAQ4WOODCRPYCVNM4FQYVBILEKNB7EHP4IPRTYCA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.423322712 (321h40m24.885415185s) [noise=true , address=77.90.31.16:4734 , piece_id=5QS6YKX3ZRX6KQWTMUGMQCUTK3ON4LCUYZUS5B2DE2UQUTHGSRUA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.425171101 (321h40m24.887263584s) [noise=true , address=77.90.31.21:4877 , piece_id=63R63JLRLK7UGVNCUOIXTFHKW3OKFHXFGUXD5D6C2RJC4MVBMJZQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.426722144 (321h40m24.888814627s) [noise=true , address=145.79.227.7:4294 , piece_id=MH7WZ6AWGTDRTHIZKSTZVR34JAZTM3HO2DGKMOXRATLLOPBVSLVQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.428244814 (321h40m24.890337297s) [noise=true , address=45.91.116.11:4118 , piece_id=KR4UNHZZZXWRI4OEPJ4ALYAPCMMWOWDHSF6DQK7FKQMZQ5WBY5IQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.429659388 (321h40m24.891751861s) [noise=true , address=77.90.31.14:4661 , piece_id=A4YLZTZCGXDYQCXSXBFIGLEXHQYNNXH6LMSA7RE6LNUTECPELKMQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.430972938 (321h40m24.893065411s) [noise=true , address=145.79.227.19:4079 , piece_id=CYJWBNPZA3WY3GF5NKFUYR2NI734LUXUP4MS2FWHZRBAE233V2PA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.431406765 (321h40m24.893499238s) [noise=true , address=77.90.31.19:4841 , piece_id=I4NNAD37P6MJX6HELBMJVOK4ZUUMIXTQUHR72TVFATV6ZUJZM2DQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:41.114941991 (321h40m26.577034464s) [upload=283103]: Next returned nil segment Oct 6 23:11:41.114961708 (321h40m26.577054191s) [upload=283103]: Waiting for error group managing segments... Oct 6 23:11:41.395880787 (321h40m26.857973271s) [noise=true , address=145.79.227.2:4115 , piece_id=CABLKD7HYRPA74BXVXKLEBTDFWAOPY66H76XYY4BAR5QOJ46DJQQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.396221086 (321h40m26.858313569s) [noise=true , address=145.79.227.19:4077 , piece_id=NMQQC6MHTFNASAQ4WOODCRPYCVNM4FQYVBILEKNB7EHP4IPRTYCA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.400339508 (321h40m26.862431981s) [noise=true , address=145.79.227.13:4802 , piece_id=SLDTYBRLPKJBWI6TH6NBW34EV3R537UEUJJZFGFYKUKKMHUYPECA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.404391554 (321h40m26.866484027s) [noise=true , address=145.79.227.6:4257 , piece_id=L5AF3ISZBP6QF3TZFBQW2MS5KM27Y4JBBONSLHO6CHJZEGQZQI4Q, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.410508050 (321h40m26.872600523s) [noise=true , address=145.79.227.7:4294 , piece_id=MH7WZ6AWGTDRTHIZKSTZVR34JAZTM3HO2DGKMOXRATLLOPBVSLVQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.422887814 (321h40m26.884980287s) [noise=true , address=145.79.227.19:4079 , piece_id=CYJWBNPZA3WY3GF5NKFUYR2NI734LUXUP4MS2FWHZRBAE233V2PA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.423758171 (321h40m26.885850644s) [noise=true , address=145.79.227.4:4185 , piece_id=G75L47ZLAM73SH4UW5HYVSJKHWUN7WYCMGH3HNPSZ67PASTAYVSA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.493892409 (321h40m26.955984892s) [noise=true , address=145.79.227.16:4911 , piece_id=KRVX3BN2PSBZSKNW3XFPKBREEJYCFWD2PJCIP7FHF6JJWYPGSNKQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:12:08.456557177 (321h40m53.91864966s) [noise=true , address=77.90.31.14:4661 , piece_id=A4YLZTZCGXDYQCXSXBFIGLEXHQYNNXH6LMSA7RE6LNUTECPELKMQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.456872789 (321h40m53.918965282s) [noise=true , address=77.90.31.13:4627 , piece_id=KRXAAGW63PKK5DY5MJZF5UPQLIPX52CKKWA3ZFH26WBTZCBE6ORQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: rpc: dial tcp 77.90.31.13:4627: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457082308 (321h40m53.919174781s) [noise=true , address=45.91.116.11:4118 , piece_id=KR4UNHZZZXWRI4OEPJ4ALYAPCMMWOWDHSF6DQK7FKQMZQ5WBY5IQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457260407 (321h40m53.91935288s) [noise=true , address=77.90.31.15:4697 , piece_id=FYHG55O4FUO6DLGJEU6MCHB3ORBRL5G6JGV2YGO3RGUDHUXE4AQA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457432174 (321h40m53.919524657s) [noise=true , address=77.90.31.16:4734 , piece_id=5QS6YKX3ZRX6KQWTMUGMQCUTK3ON4LCUYZUS5B2DE2UQUTHGSRUA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457616485 (321h40m53.919708968s) [noise=true , address=77.90.31.19:4841 , piece_id=I4NNAD37P6MJX6HELBMJVOK4ZUUMIXTQUHR72TVFATV6ZUJZM2DQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 188.166.225.170:45634->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457895887 (321h40m53.91998837s) [noise=true , address=77.90.31.11:4553 , piece_id=IOOB6KDBP32D4WIHQ3JCWBFUO7ITOIJMYQBQEQ6QZV2CWURHM6IA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 188.166.225.170:42640->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.458065830 (321h40m53.920158304s) [noise=true , address=77.90.31.21:4877 , piece_id=63R63JLRLK7UGVNCUOIXTFHKW3OKFHXFGUXD5D6C2RJC4MVBMJZQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:42068->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.458094555 (321h40m53.920187038s) [seg_pos={0 0}, upload=283103]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 6 23:12:08.698421029 (321h40m54.160513512s) [upload=283103]: Done uploading segments... Oct 6 23:12:08.698456296 (321h40m54.160548779s) [upload=283103]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.286608251s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XX........X.................] duration: 7.197823389s FAIL (2025-10-06 23:13:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.197851022s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X..................X......X.] duration: 7.460717393s FAIL (2025-10-06 23:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 7400762s) [noise=true , address=145.79.227.31:4820 , piece_id=KDF3S2SFAHERKYTZXLR4UPOOXPZ475ORJCO7S3BOJGMW3NKBY4GQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.412288768 (321h40m20.874381241s) [noise=true , address=77.90.31.18:4831 , piece_id=IKUBFGKWHIKFHVSPAKISHQ2A3FQOQTWD3YVNKXLOREBL4KPFZSYQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.412644696 (321h40m20.874737169s) [noise=true , address=64.49.13.137:4502 , piece_id=W7SGEHG7ZLLYMRM4Z4XZ3CTSZJJDM6BGUV6IXOC7VXEC7YPLUG2Q, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413017876 (321h40m20.875110359s) [noise=true , address=45.91.116.7:4027 , piece_id=UEA4UJLNVUTWSNL6NUUN7SLCHY2BZ7SC6NQAHNCSMSQX474OL2JQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413396136 (321h40m20.875488619s) [noise=true , address=77.90.31.19:4851 , piece_id=53XK564AIKRQUC5CIVJLDPPMCKLA46WNSTAPJX46QJHJQ6QMEAFQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413895047 (321h40m20.87598752s) [noise=true , address=45.91.116.11:4129 , piece_id=KIAPVZXQ4QSEEAHXA5JCBKNJOS5D77B336O6QEFAABYI2WCBEFQA, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.414559813 (321h40m20.876652296s) [noise=true , address=145.79.227.29:4943 , piece_id=LAOUJXCAYRCVWU3VDX76FPB5F3HOYZEA3YVDT4F5KE4UG2OMZB6Q, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415078200 (321h40m20.877170683s) [noise=true , address=77.90.31.19:4869 , piece_id=WSJXCO7K2TG6WYVT6GCSJBWNKCE2YRT4DYEGFMJZPVGZ7UDXDLYQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415439157 (321h40m20.87753163s) [noise=true , address=45.91.116.12:4241 , piece_id=CEDAYVITA66N2QA2KOCK7KIU3KBBYYS2VXWO4GPEWSVP7Z4RULNA, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415763504 (321h40m20.877855987s) [noise=true , address=45.91.116.13:4166 , piece_id=7QTSHYGDBUTFHUQDFNJYQYXN4V6IRU7IMQ6IFMKBFYVC3XDXICPQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.416122719 (321h40m20.878215202s) [noise=true , address=77.90.31.11:4579 , piece_id=NE4RUF4T6U7UOM4AMCVJNOOIQ5O5BHETJ3YZCGVG7EI44SQUSMFQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:36.702309353 (321h40m22.164401836s) [upload=283102]: Next returned nil segment Oct 6 23:11:36.702329182 (321h40m22.164421665s) [upload=283102]: Waiting for error group managing segments... Oct 6 23:11:37.029061628 (321h40m22.491154101s) [noise=true , address=64.49.13.137:4502 , piece_id=W7SGEHG7ZLLYMRM4Z4XZ3CTSZJJDM6BGUV6IXOC7VXEC7YPLUG2Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.252437915 (321h40m22.714530398s) [noise=true , address=145.79.227.34:4929 , piece_id=NHKA33B6AQLEIRTLYQACKESLPOSM7WOABQOQDEBXUQYKDNXMDUCQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.256976217 (321h40m22.7190687s) [noise=true , address=145.79.227.21:4479 , piece_id=27LBIQRR5KL5MQGWWAMXQ3NWAYGRJT6RIELLLPFURPI6MMBVGMBA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.257492219 (321h40m22.719584692s) [noise=true , address=145.79.227.31:4820 , piece_id=KDF3S2SFAHERKYTZXLR4UPOOXPZ475ORJCO7S3BOJGMW3NKBY4GQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.277387253 (321h40m22.739479726s) [noise=true , address=145.79.227.29:4943 , piece_id=LAOUJXCAYRCVWU3VDX76FPB5F3HOYZEA3YVDT4F5KE4UG2OMZB6Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:12:34.456871161 (321h41m19.918963654s) [noise=true , address=77.90.31.19:4869 , piece_id=WSJXCO7K2TG6WYVT6GCSJBWNKCE2YRT4DYEGFMJZPVGZ7UDXDLYQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: rpc: dial tcp 77.90.31.19:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.456947426 (321h41m19.919039909s) [noise=true , address=77.90.31.19:4851 , piece_id=53XK564AIKRQUC5CIVJLDPPMCKLA46WNSTAPJX46QJHJQ6QMEAFQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457178967 (321h41m19.91927145s) [noise=true , address=77.90.31.11:4579 , piece_id=NE4RUF4T6U7UOM4AMCVJNOOIQ5O5BHETJ3YZCGVG7EI44SQUSMFQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: read tcp 188.166.225.170:58086->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457193044 (321h41m19.919285527s) [noise=true , address=45.91.116.6:4988 , piece_id=A2P6YDUI3S57WSUK7QJST6LXXNHDDUYF542MS7LU6OQF723J2QIQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS, address:45.91.116.6:4988): protocol: rpc: dial tcp 45.91.116.6:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457329935 (321h41m19.919422418s) [noise=true , address=45.91.116.5:4309 , piece_id=DKBEWGUHKP6IIVHO7S7SM2O2UXPVXQHJT6Q7GXJJNOZXWRRJA7KA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1CHPuJdsjXtHCho9WdqpSvzRbEJqmAXchTkhmXgVqWQTTTCrV3, address:45.91.116.5:4309): protocol: rpc: dial tcp 45.91.116.5:4309: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457367937 (321h41m19.91946042s) [noise=true , address=45.91.116.14:4195 , piece_id=NZ3BQUPMWERJ3J45MXI6WZOCATOLVZ7VMNULANLYHQBDRS4E657A, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457501181 (321h41m19.919593654s) [noise=true , address=45.91.116.7:4027 , piece_id=UEA4UJLNVUTWSNL6NUUN7SLCHY2BZ7SC6NQAHNCSMSQX474OL2JQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq, address:45.91.116.7:4027): protocol: write tcp 188.166.225.170:56512->45.91.116.7:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457509076 (321h41m19.919601569s) [noise=true , address=45.91.116.12:4241 , piece_id=CEDAYVITA66N2QA2KOCK7KIU3KBBYYS2VXWO4GPEWSVP7Z4RULNA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:128wA5sH9sQDinHwwiTGL7WDktf6yD8EqQqy9mCxeZAKorYgsXD, address:45.91.116.12:4241): protocol: read tcp 188.166.225.170:37074->45.91.116.12:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457609828 (321h41m19.919702301s) [noise=true , address=45.91.116.11:4129 , piece_id=KIAPVZXQ4QSEEAHXA5JCBKNJOS5D77B336O6QEFAABYI2WCBEFQA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: read tcp 188.166.225.170:36136->45.91.116.11:4129: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457634866 (321h41m19.919727349s) [noise=true , address=77.90.31.13:4636 , piece_id=EVMVXR62UI42REJ6GHRYY5EANJARGC3LZYPBDFSOIDS6AO72KQWQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: read tcp 188.166.225.170:37498->77.90.31.13:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457770924 (321h41m19.919863397s) [noise=true , address=77.90.31.21:4907 , piece_id=F64NQLEYP2CYXNBH7YT3EXGR35OCH2RPXN7M2F46SSHGIDE5NW5Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: read tcp 188.166.225.170:37048->77.90.31.21:4907: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457900290 (321h41m19.919992773s) [noise=true , address=77.90.31.18:4831 , piece_id=IKUBFGKWHIKFHVSPAKISHQ2A3FQOQTWD3YVNKXLOREBL4KPFZSYQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: read tcp 188.166.225.170:59108->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457935207 (321h41m19.9200277s) [noise=true , address=45.91.116.13:4166 , piece_id=7QTSHYGDBUTFHUQDFNJYQYXN4V6IRU7IMQ6IFMKBFYVC3XDXICPQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: read tcp 188.166.225.170:34944->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457964232 (321h41m19.920056725s) [seg_pos={0 0}, upload=283102]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Oct 6 23:12:34.699771335 (321h41m20.161863829s) [upload=283102]: Done uploading segments... Oct 6 23:12:34.699827862 (321h41m20.161920356s) [upload=283102]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 7.460745637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................X...........XX.......X..X..X.] duration: 7.959226368s FAIL (2025-10-06 23:12:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ) [upload=283103]: Done starting upload Oct 6 23:11:39.165713278 (321h40m24.627805761s) [upload=283103]: Uploading segments... Oct 6 23:11:39.165926464 (321h40m24.628018957s) [upload=283103]: Got next segment. Inline: false Oct 6 23:11:39.166463165 (321h40m24.628555638s) [upload=283103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:11:39.166468907 (321h40m24.62856139s) [upload=283103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:11:39.416464112 (321h40m24.878556585s) [seg_pos={0 0}, upload=283103]: Begin upload segment... Oct 6 23:11:39.416533804 (321h40m24.878626277s) [seg_pos={0 0}, upload=283103]: Done begin upload segment. Oct 6 23:11:39.416562188 (321h40m24.878654661s) [noise=true , address=145.79.227.4:4185 , piece_id=G75L47ZLAM73SH4UW5HYVSJKHWUN7WYCMGH3HNPSZ67PASTAYVSA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.416623334 (321h40m24.878715817s) [noise=true , address=145.79.227.13:4802 , piece_id=SLDTYBRLPKJBWI6TH6NBW34EV3R537UEUJJZFGFYKUKKMHUYPECA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.417313729 (321h40m24.879406212s) [noise=true , address=77.90.31.13:4627 , piece_id=KRXAAGW63PKK5DY5MJZF5UPQLIPX52CKKWA3ZFH26WBTZCBE6ORQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.417781509 (321h40m24.879874002s) [noise=true , address=145.79.227.2:4115 , piece_id=CABLKD7HYRPA74BXVXKLEBTDFWAOPY66H76XYY4BAR5QOJ46DJQQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.418910009 (321h40m24.881002482s) [noise=true , address=145.79.227.16:4911 , piece_id=KRVX3BN2PSBZSKNW3XFPKBREEJYCFWD2PJCIP7FHF6JJWYPGSNKQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.419368221 (321h40m24.881460704s) [noise=true , address=145.79.227.6:4257 , piece_id=L5AF3ISZBP6QF3TZFBQW2MS5KM27Y4JBBONSLHO6CHJZEGQZQI4Q, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.420810096 (321h40m24.882902569s) [noise=true , address=77.90.31.11:4553 , piece_id=IOOB6KDBP32D4WIHQ3JCWBFUO7ITOIJMYQBQEQ6QZV2CWURHM6IA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.421327622 (321h40m24.883420105s) [noise=true , address=77.90.31.15:4697 , piece_id=FYHG55O4FUO6DLGJEU6MCHB3ORBRL5G6JGV2YGO3RGUDHUXE4AQA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.422674918 (321h40m24.884767401s) [noise=true , address=145.79.227.19:4077 , piece_id=NMQQC6MHTFNASAQ4WOODCRPYCVNM4FQYVBILEKNB7EHP4IPRTYCA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.423322712 (321h40m24.885415185s) [noise=true , address=77.90.31.16:4734 , piece_id=5QS6YKX3ZRX6KQWTMUGMQCUTK3ON4LCUYZUS5B2DE2UQUTHGSRUA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.425171101 (321h40m24.887263584s) [noise=true , address=77.90.31.21:4877 , piece_id=63R63JLRLK7UGVNCUOIXTFHKW3OKFHXFGUXD5D6C2RJC4MVBMJZQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.426722144 (321h40m24.888814627s) [noise=true , address=145.79.227.7:4294 , piece_id=MH7WZ6AWGTDRTHIZKSTZVR34JAZTM3HO2DGKMOXRATLLOPBVSLVQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.428244814 (321h40m24.890337297s) [noise=true , address=45.91.116.11:4118 , piece_id=KR4UNHZZZXWRI4OEPJ4ALYAPCMMWOWDHSF6DQK7FKQMZQ5WBY5IQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.429659388 (321h40m24.891751861s) [noise=true , address=77.90.31.14:4661 , piece_id=A4YLZTZCGXDYQCXSXBFIGLEXHQYNNXH6LMSA7RE6LNUTECPELKMQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.430972938 (321h40m24.893065411s) [noise=true , address=145.79.227.19:4079 , piece_id=CYJWBNPZA3WY3GF5NKFUYR2NI734LUXUP4MS2FWHZRBAE233V2PA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.431406765 (321h40m24.893499238s) [noise=true , address=77.90.31.19:4841 , piece_id=I4NNAD37P6MJX6HELBMJVOK4ZUUMIXTQUHR72TVFATV6ZUJZM2DQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:41.114941991 (321h40m26.577034464s) [upload=283103]: Next returned nil segment Oct 6 23:11:41.114961708 (321h40m26.577054191s) [upload=283103]: Waiting for error group managing segments... Oct 6 23:11:41.395880787 (321h40m26.857973271s) [noise=true , address=145.79.227.2:4115 , piece_id=CABLKD7HYRPA74BXVXKLEBTDFWAOPY66H76XYY4BAR5QOJ46DJQQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.396221086 (321h40m26.858313569s) [noise=true , address=145.79.227.19:4077 , piece_id=NMQQC6MHTFNASAQ4WOODCRPYCVNM4FQYVBILEKNB7EHP4IPRTYCA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.400339508 (321h40m26.862431981s) [noise=true , address=145.79.227.13:4802 , piece_id=SLDTYBRLPKJBWI6TH6NBW34EV3R537UEUJJZFGFYKUKKMHUYPECA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.404391554 (321h40m26.866484027s) [noise=true , address=145.79.227.6:4257 , piece_id=L5AF3ISZBP6QF3TZFBQW2MS5KM27Y4JBBONSLHO6CHJZEGQZQI4Q, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.410508050 (321h40m26.872600523s) [noise=true , address=145.79.227.7:4294 , piece_id=MH7WZ6AWGTDRTHIZKSTZVR34JAZTM3HO2DGKMOXRATLLOPBVSLVQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.422887814 (321h40m26.884980287s) [noise=true , address=145.79.227.19:4079 , piece_id=CYJWBNPZA3WY3GF5NKFUYR2NI734LUXUP4MS2FWHZRBAE233V2PA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.423758171 (321h40m26.885850644s) [noise=true , address=145.79.227.4:4185 , piece_id=G75L47ZLAM73SH4UW5HYVSJKHWUN7WYCMGH3HNPSZ67PASTAYVSA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.493892409 (321h40m26.955984892s) [noise=true , address=145.79.227.16:4911 , piece_id=KRVX3BN2PSBZSKNW3XFPKBREEJYCFWD2PJCIP7FHF6JJWYPGSNKQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:12:08.456557177 (321h40m53.91864966s) [noise=true , address=77.90.31.14:4661 , piece_id=A4YLZTZCGXDYQCXSXBFIGLEXHQYNNXH6LMSA7RE6LNUTECPELKMQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.456872789 (321h40m53.918965282s) [noise=true , address=77.90.31.13:4627 , piece_id=KRXAAGW63PKK5DY5MJZF5UPQLIPX52CKKWA3ZFH26WBTZCBE6ORQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: rpc: dial tcp 77.90.31.13:4627: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457082308 (321h40m53.919174781s) [noise=true , address=45.91.116.11:4118 , piece_id=KR4UNHZZZXWRI4OEPJ4ALYAPCMMWOWDHSF6DQK7FKQMZQ5WBY5IQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457260407 (321h40m53.91935288s) [noise=true , address=77.90.31.15:4697 , piece_id=FYHG55O4FUO6DLGJEU6MCHB3ORBRL5G6JGV2YGO3RGUDHUXE4AQA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457432174 (321h40m53.919524657s) [noise=true , address=77.90.31.16:4734 , piece_id=5QS6YKX3ZRX6KQWTMUGMQCUTK3ON4LCUYZUS5B2DE2UQUTHGSRUA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457616485 (321h40m53.919708968s) [noise=true , address=77.90.31.19:4841 , piece_id=I4NNAD37P6MJX6HELBMJVOK4ZUUMIXTQUHR72TVFATV6ZUJZM2DQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 188.166.225.170:45634->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457895887 (321h40m53.91998837s) [noise=true , address=77.90.31.11:4553 , piece_id=IOOB6KDBP32D4WIHQ3JCWBFUO7ITOIJMYQBQEQ6QZV2CWURHM6IA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 188.166.225.170:42640->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.458065830 (321h40m53.920158304s) [noise=true , address=77.90.31.21:4877 , piece_id=63R63JLRLK7UGVNCUOIXTFHKW3OKFHXFGUXD5D6C2RJC4MVBMJZQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:42068->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.458094555 (321h40m53.920187038s) [seg_pos={0 0}, upload=283103]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 6 23:12:08.698421029 (321h40m54.160513512s) [upload=283103]: Done uploading segments... Oct 6 23:12:08.698456296 (321h40m54.160548779s) [upload=283103]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.959277405s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.926613903s FAIL (2025-10-06 23:12:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.926631027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..................X......X] duration: 1m0.245529287s FAIL (2025-10-06 23:12:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 7400762s) [noise=true , address=145.79.227.31:4820 , piece_id=KDF3S2SFAHERKYTZXLR4UPOOXPZ475ORJCO7S3BOJGMW3NKBY4GQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.412288768 (321h40m20.874381241s) [noise=true , address=77.90.31.18:4831 , piece_id=IKUBFGKWHIKFHVSPAKISHQ2A3FQOQTWD3YVNKXLOREBL4KPFZSYQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.412644696 (321h40m20.874737169s) [noise=true , address=64.49.13.137:4502 , piece_id=W7SGEHG7ZLLYMRM4Z4XZ3CTSZJJDM6BGUV6IXOC7VXEC7YPLUG2Q, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413017876 (321h40m20.875110359s) [noise=true , address=45.91.116.7:4027 , piece_id=UEA4UJLNVUTWSNL6NUUN7SLCHY2BZ7SC6NQAHNCSMSQX474OL2JQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413396136 (321h40m20.875488619s) [noise=true , address=77.90.31.19:4851 , piece_id=53XK564AIKRQUC5CIVJLDPPMCKLA46WNSTAPJX46QJHJQ6QMEAFQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.413895047 (321h40m20.87598752s) [noise=true , address=45.91.116.11:4129 , piece_id=KIAPVZXQ4QSEEAHXA5JCBKNJOS5D77B336O6QEFAABYI2WCBEFQA, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.414559813 (321h40m20.876652296s) [noise=true , address=145.79.227.29:4943 , piece_id=LAOUJXCAYRCVWU3VDX76FPB5F3HOYZEA3YVDT4F5KE4UG2OMZB6Q, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415078200 (321h40m20.877170683s) [noise=true , address=77.90.31.19:4869 , piece_id=WSJXCO7K2TG6WYVT6GCSJBWNKCE2YRT4DYEGFMJZPVGZ7UDXDLYQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415439157 (321h40m20.87753163s) [noise=true , address=45.91.116.12:4241 , piece_id=CEDAYVITA66N2QA2KOCK7KIU3KBBYYS2VXWO4GPEWSVP7Z4RULNA, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.415763504 (321h40m20.877855987s) [noise=true , address=45.91.116.13:4166 , piece_id=7QTSHYGDBUTFHUQDFNJYQYXN4V6IRU7IMQ6IFMKBFYVC3XDXICPQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:35.416122719 (321h40m20.878215202s) [noise=true , address=77.90.31.11:4579 , piece_id=NE4RUF4T6U7UOM4AMCVJNOOIQ5O5BHETJ3YZCGVG7EI44SQUSMFQ, seg_pos={0 0}, upload=283102]: Uploading piece... Oct 6 23:11:36.702309353 (321h40m22.164401836s) [upload=283102]: Next returned nil segment Oct 6 23:11:36.702329182 (321h40m22.164421665s) [upload=283102]: Waiting for error group managing segments... Oct 6 23:11:37.029061628 (321h40m22.491154101s) [noise=true , address=64.49.13.137:4502 , piece_id=W7SGEHG7ZLLYMRM4Z4XZ3CTSZJJDM6BGUV6IXOC7VXEC7YPLUG2Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.252437915 (321h40m22.714530398s) [noise=true , address=145.79.227.34:4929 , piece_id=NHKA33B6AQLEIRTLYQACKESLPOSM7WOABQOQDEBXUQYKDNXMDUCQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.256976217 (321h40m22.7190687s) [noise=true , address=145.79.227.21:4479 , piece_id=27LBIQRR5KL5MQGWWAMXQ3NWAYGRJT6RIELLLPFURPI6MMBVGMBA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.257492219 (321h40m22.719584692s) [noise=true , address=145.79.227.31:4820 , piece_id=KDF3S2SFAHERKYTZXLR4UPOOXPZ475ORJCO7S3BOJGMW3NKBY4GQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:11:37.277387253 (321h40m22.739479726s) [noise=true , address=145.79.227.29:4943 , piece_id=LAOUJXCAYRCVWU3VDX76FPB5F3HOYZEA3YVDT4F5KE4UG2OMZB6Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: <nil> Oct 6 23:12:34.456871161 (321h41m19.918963654s) [noise=true , address=77.90.31.19:4869 , piece_id=WSJXCO7K2TG6WYVT6GCSJBWNKCE2YRT4DYEGFMJZPVGZ7UDXDLYQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: rpc: dial tcp 77.90.31.19:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.456947426 (321h41m19.919039909s) [noise=true , address=77.90.31.19:4851 , piece_id=53XK564AIKRQUC5CIVJLDPPMCKLA46WNSTAPJX46QJHJQ6QMEAFQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457178967 (321h41m19.91927145s) [noise=true , address=77.90.31.11:4579 , piece_id=NE4RUF4T6U7UOM4AMCVJNOOIQ5O5BHETJ3YZCGVG7EI44SQUSMFQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: read tcp 188.166.225.170:58086->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457193044 (321h41m19.919285527s) [noise=true , address=45.91.116.6:4988 , piece_id=A2P6YDUI3S57WSUK7QJST6LXXNHDDUYF542MS7LU6OQF723J2QIQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS, address:45.91.116.6:4988): protocol: rpc: dial tcp 45.91.116.6:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457329935 (321h41m19.919422418s) [noise=true , address=45.91.116.5:4309 , piece_id=DKBEWGUHKP6IIVHO7S7SM2O2UXPVXQHJT6Q7GXJJNOZXWRRJA7KA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1CHPuJdsjXtHCho9WdqpSvzRbEJqmAXchTkhmXgVqWQTTTCrV3, address:45.91.116.5:4309): protocol: rpc: dial tcp 45.91.116.5:4309: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457367937 (321h41m19.91946042s) [noise=true , address=45.91.116.14:4195 , piece_id=NZ3BQUPMWERJ3J45MXI6WZOCATOLVZ7VMNULANLYHQBDRS4E657A, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457501181 (321h41m19.919593654s) [noise=true , address=45.91.116.7:4027 , piece_id=UEA4UJLNVUTWSNL6NUUN7SLCHY2BZ7SC6NQAHNCSMSQX474OL2JQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq, address:45.91.116.7:4027): protocol: write tcp 188.166.225.170:56512->45.91.116.7:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457509076 (321h41m19.919601569s) [noise=true , address=45.91.116.12:4241 , piece_id=CEDAYVITA66N2QA2KOCK7KIU3KBBYYS2VXWO4GPEWSVP7Z4RULNA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:128wA5sH9sQDinHwwiTGL7WDktf6yD8EqQqy9mCxeZAKorYgsXD, address:45.91.116.12:4241): protocol: read tcp 188.166.225.170:37074->45.91.116.12:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457609828 (321h41m19.919702301s) [noise=true , address=45.91.116.11:4129 , piece_id=KIAPVZXQ4QSEEAHXA5JCBKNJOS5D77B336O6QEFAABYI2WCBEFQA, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: read tcp 188.166.225.170:36136->45.91.116.11:4129: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457634866 (321h41m19.919727349s) [noise=true , address=77.90.31.13:4636 , piece_id=EVMVXR62UI42REJ6GHRYY5EANJARGC3LZYPBDFSOIDS6AO72KQWQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: read tcp 188.166.225.170:37498->77.90.31.13:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457770924 (321h41m19.919863397s) [noise=true , address=77.90.31.21:4907 , piece_id=F64NQLEYP2CYXNBH7YT3EXGR35OCH2RPXN7M2F46SSHGIDE5NW5Q, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: read tcp 188.166.225.170:37048->77.90.31.21:4907: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457900290 (321h41m19.919992773s) [noise=true , address=77.90.31.18:4831 , piece_id=IKUBFGKWHIKFHVSPAKISHQ2A3FQOQTWD3YVNKXLOREBL4KPFZSYQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: read tcp 188.166.225.170:59108->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457935207 (321h41m19.9200277s) [noise=true , address=45.91.116.13:4166 , piece_id=7QTSHYGDBUTFHUQDFNJYQYXN4V6IRU7IMQ6IFMKBFYVC3XDXICPQ, seg_pos={0 0}, upload=283102]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: read tcp 188.166.225.170:34944->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:34.457964232 (321h41m19.920056725s) [seg_pos={0 0}, upload=283102]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Oct 6 23:12:34.699771335 (321h41m20.161863829s) [upload=283102]: Done uploading segments... Oct 6 23:12:34.699827862 (321h41m20.161920356s) [upload=283102]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.245555596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XX........X................] duration: 11.537037905s FAIL (2025-10-06 23:12:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.537066019s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000975382s FAIL (2025-10-06 23:12:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000992194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................X...........XX.......X..X..X] duration: 30.243288442s FAIL (2025-10-06 23:12:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ) [upload=283103]: Done starting upload Oct 6 23:11:39.165713278 (321h40m24.627805761s) [upload=283103]: Uploading segments... Oct 6 23:11:39.165926464 (321h40m24.628018957s) [upload=283103]: Got next segment. Inline: false Oct 6 23:11:39.166463165 (321h40m24.628555638s) [upload=283103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:11:39.166468907 (321h40m24.62856139s) [upload=283103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:11:39.416464112 (321h40m24.878556585s) [seg_pos={0 0}, upload=283103]: Begin upload segment... Oct 6 23:11:39.416533804 (321h40m24.878626277s) [seg_pos={0 0}, upload=283103]: Done begin upload segment. Oct 6 23:11:39.416562188 (321h40m24.878654661s) [noise=true , address=145.79.227.4:4185 , piece_id=G75L47ZLAM73SH4UW5HYVSJKHWUN7WYCMGH3HNPSZ67PASTAYVSA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.416623334 (321h40m24.878715817s) [noise=true , address=145.79.227.13:4802 , piece_id=SLDTYBRLPKJBWI6TH6NBW34EV3R537UEUJJZFGFYKUKKMHUYPECA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.417313729 (321h40m24.879406212s) [noise=true , address=77.90.31.13:4627 , piece_id=KRXAAGW63PKK5DY5MJZF5UPQLIPX52CKKWA3ZFH26WBTZCBE6ORQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.417781509 (321h40m24.879874002s) [noise=true , address=145.79.227.2:4115 , piece_id=CABLKD7HYRPA74BXVXKLEBTDFWAOPY66H76XYY4BAR5QOJ46DJQQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.418910009 (321h40m24.881002482s) [noise=true , address=145.79.227.16:4911 , piece_id=KRVX3BN2PSBZSKNW3XFPKBREEJYCFWD2PJCIP7FHF6JJWYPGSNKQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.419368221 (321h40m24.881460704s) [noise=true , address=145.79.227.6:4257 , piece_id=L5AF3ISZBP6QF3TZFBQW2MS5KM27Y4JBBONSLHO6CHJZEGQZQI4Q, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.420810096 (321h40m24.882902569s) [noise=true , address=77.90.31.11:4553 , piece_id=IOOB6KDBP32D4WIHQ3JCWBFUO7ITOIJMYQBQEQ6QZV2CWURHM6IA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.421327622 (321h40m24.883420105s) [noise=true , address=77.90.31.15:4697 , piece_id=FYHG55O4FUO6DLGJEU6MCHB3ORBRL5G6JGV2YGO3RGUDHUXE4AQA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.422674918 (321h40m24.884767401s) [noise=true , address=145.79.227.19:4077 , piece_id=NMQQC6MHTFNASAQ4WOODCRPYCVNM4FQYVBILEKNB7EHP4IPRTYCA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.423322712 (321h40m24.885415185s) [noise=true , address=77.90.31.16:4734 , piece_id=5QS6YKX3ZRX6KQWTMUGMQCUTK3ON4LCUYZUS5B2DE2UQUTHGSRUA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.425171101 (321h40m24.887263584s) [noise=true , address=77.90.31.21:4877 , piece_id=63R63JLRLK7UGVNCUOIXTFHKW3OKFHXFGUXD5D6C2RJC4MVBMJZQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.426722144 (321h40m24.888814627s) [noise=true , address=145.79.227.7:4294 , piece_id=MH7WZ6AWGTDRTHIZKSTZVR34JAZTM3HO2DGKMOXRATLLOPBVSLVQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.428244814 (321h40m24.890337297s) [noise=true , address=45.91.116.11:4118 , piece_id=KR4UNHZZZXWRI4OEPJ4ALYAPCMMWOWDHSF6DQK7FKQMZQ5WBY5IQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.429659388 (321h40m24.891751861s) [noise=true , address=77.90.31.14:4661 , piece_id=A4YLZTZCGXDYQCXSXBFIGLEXHQYNNXH6LMSA7RE6LNUTECPELKMQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.430972938 (321h40m24.893065411s) [noise=true , address=145.79.227.19:4079 , piece_id=CYJWBNPZA3WY3GF5NKFUYR2NI734LUXUP4MS2FWHZRBAE233V2PA, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:39.431406765 (321h40m24.893499238s) [noise=true , address=77.90.31.19:4841 , piece_id=I4NNAD37P6MJX6HELBMJVOK4ZUUMIXTQUHR72TVFATV6ZUJZM2DQ, seg_pos={0 0}, upload=283103]: Uploading piece... Oct 6 23:11:41.114941991 (321h40m26.577034464s) [upload=283103]: Next returned nil segment Oct 6 23:11:41.114961708 (321h40m26.577054191s) [upload=283103]: Waiting for error group managing segments... Oct 6 23:11:41.395880787 (321h40m26.857973271s) [noise=true , address=145.79.227.2:4115 , piece_id=CABLKD7HYRPA74BXVXKLEBTDFWAOPY66H76XYY4BAR5QOJ46DJQQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.396221086 (321h40m26.858313569s) [noise=true , address=145.79.227.19:4077 , piece_id=NMQQC6MHTFNASAQ4WOODCRPYCVNM4FQYVBILEKNB7EHP4IPRTYCA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.400339508 (321h40m26.862431981s) [noise=true , address=145.79.227.13:4802 , piece_id=SLDTYBRLPKJBWI6TH6NBW34EV3R537UEUJJZFGFYKUKKMHUYPECA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.404391554 (321h40m26.866484027s) [noise=true , address=145.79.227.6:4257 , piece_id=L5AF3ISZBP6QF3TZFBQW2MS5KM27Y4JBBONSLHO6CHJZEGQZQI4Q, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.410508050 (321h40m26.872600523s) [noise=true , address=145.79.227.7:4294 , piece_id=MH7WZ6AWGTDRTHIZKSTZVR34JAZTM3HO2DGKMOXRATLLOPBVSLVQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.422887814 (321h40m26.884980287s) [noise=true , address=145.79.227.19:4079 , piece_id=CYJWBNPZA3WY3GF5NKFUYR2NI734LUXUP4MS2FWHZRBAE233V2PA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.423758171 (321h40m26.885850644s) [noise=true , address=145.79.227.4:4185 , piece_id=G75L47ZLAM73SH4UW5HYVSJKHWUN7WYCMGH3HNPSZ67PASTAYVSA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:11:41.493892409 (321h40m26.955984892s) [noise=true , address=145.79.227.16:4911 , piece_id=KRVX3BN2PSBZSKNW3XFPKBREEJYCFWD2PJCIP7FHF6JJWYPGSNKQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: <nil> Oct 6 23:12:08.456557177 (321h40m53.91864966s) [noise=true , address=77.90.31.14:4661 , piece_id=A4YLZTZCGXDYQCXSXBFIGLEXHQYNNXH6LMSA7RE6LNUTECPELKMQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.456872789 (321h40m53.918965282s) [noise=true , address=77.90.31.13:4627 , piece_id=KRXAAGW63PKK5DY5MJZF5UPQLIPX52CKKWA3ZFH26WBTZCBE6ORQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: rpc: dial tcp 77.90.31.13:4627: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457082308 (321h40m53.919174781s) [noise=true , address=45.91.116.11:4118 , piece_id=KR4UNHZZZXWRI4OEPJ4ALYAPCMMWOWDHSF6DQK7FKQMZQ5WBY5IQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457260407 (321h40m53.91935288s) [noise=true , address=77.90.31.15:4697 , piece_id=FYHG55O4FUO6DLGJEU6MCHB3ORBRL5G6JGV2YGO3RGUDHUXE4AQA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457432174 (321h40m53.919524657s) [noise=true , address=77.90.31.16:4734 , piece_id=5QS6YKX3ZRX6KQWTMUGMQCUTK3ON4LCUYZUS5B2DE2UQUTHGSRUA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457616485 (321h40m53.919708968s) [noise=true , address=77.90.31.19:4841 , piece_id=I4NNAD37P6MJX6HELBMJVOK4ZUUMIXTQUHR72TVFATV6ZUJZM2DQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 188.166.225.170:45634->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.457895887 (321h40m53.91998837s) [noise=true , address=77.90.31.11:4553 , piece_id=IOOB6KDBP32D4WIHQ3JCWBFUO7ITOIJMYQBQEQ6QZV2CWURHM6IA, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 188.166.225.170:42640->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.458065830 (321h40m53.920158304s) [noise=true , address=77.90.31.21:4877 , piece_id=63R63JLRLK7UGVNCUOIXTFHKW3OKFHXFGUXD5D6C2RJC4MVBMJZQ, seg_pos={0 0}, upload=283103]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:42068->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:12:08.458094555 (321h40m53.920187038s) [seg_pos={0 0}, upload=283103]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 6 23:12:08.698421029 (321h40m54.160513512s) [upload=283103]: Done uploading segments... Oct 6 23:12:08.698456296 (321h40m54.160548779s) [upload=283103]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.243319001s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................XX........X...............] duration: 6.166718856s FAIL (2025-10-06 23:11:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.166748713s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..................X......] duration: 6.469330361s FAIL (2025-10-06 23:11:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 21h34m30.318324337s) [upload=283020]: Got next segment. Inline: false Oct 6 23:05:44.856263134 (321h34m30.318355617s) [upload=283020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:44.856268845 (321h34m30.318361328s) [upload=283020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:45.108705808 (321h34m30.570798291s) [seg_pos={0 0}, upload=283020]: Begin upload segment... Oct 6 23:05:45.108853299 (321h34m30.570945772s) [seg_pos={0 0}, upload=283020]: Done begin upload segment. Oct 6 23:05:45.109033753 (321h34m30.571126226s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109109477 (321h34m30.571213192s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109722314 (321h34m30.571814797s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110354308 (321h34m30.572446791s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110950954 (321h34m30.573043437s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.111551888 (321h34m30.573644371s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112154475 (321h34m30.574246958s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112932658 (321h34m30.575025141s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.113602423 (321h34m30.575694896s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.114401965 (321h34m30.576494448s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.116415860 (321h34m30.578508333s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.118906663 (321h34m30.580999136s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.121067647 (321h34m30.58316012s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.129477842 (321h34m30.591570325s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.130084146 (321h34m30.592176629s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.131174373 (321h34m30.593266866s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.133360686 (321h34m30.595453169s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.135350114 (321h34m30.597442597s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:46.574429033 (321h34m32.036521506s) [upload=283020]: Next returned nil segment Oct 6 23:05:46.574445103 (321h34m32.036537576s) [upload=283020]: Waiting for error group managing segments... Oct 6 23:05:46.946795634 (321h34m32.408888107s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.970438174 (321h34m32.432530647s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.971841325 (321h34m32.433933808s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.982490243 (321h34m32.444582726s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.990921268 (321h34m32.453013741s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.993986244 (321h34m32.456078727s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.998097392 (321h34m32.460189875s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:24.533854514 (321h35m9.995946987s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.415951231 (321h35m10.878043704s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.820324670 (321h35m11.282417143s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:27.866791732 (321h35m13.328884215s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:28.427216598 (321h35m13.889309071s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:35.312697524 (321h35m20.774789997s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:44.135928415 (321h35m29.598020888s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1wR3ntox5QJSB1n45AmvQ94C6yzgihoBVYPMjsrgzTtxzhYHy1, address:45.91.116.4:4951): protocol: rpc: dial tcp 45.91.116.4:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136059113 (321h35m29.598151596s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: read tcp 188.166.225.170:41850->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136082528 (321h35m29.598175001s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: rpc: dial tcp 77.90.31.19:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136120630 (321h35m29.598213103s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX, address:45.91.116.6:4004): protocol: rpc: dial tcp 45.91.116.6:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136148143 (321h35m29.598240616s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: read tcp 188.166.225.170:47654->77.90.31.14:4669: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136173562 (321h35m29.598266035s) [seg_pos={0 0}, upload=283020]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 23:06:44.427092462 (321h35m29.889184945s) [upload=283020]: Done uploading segments... Oct 6 23:06:44.427128952 (321h35m29.889221435s) [upload=283020]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.469359958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................XX........X..............] duration: 6.424770832s FAIL (2025-10-06 23:11:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.42479082s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X...........XX.......X..X..] duration: 6.55416464s FAIL (2025-10-06 23:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 0]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:08:40.607730747 (321h37m26.06982322s) [seg_pos={0 0}, upload=283060]: Begin upload segment... Oct 6 23:08:40.607788537 (321h37m26.06988101s) [seg_pos={0 0}, upload=283060]: Done begin upload segment. Oct 6 23:08:40.607809947 (321h37m26.06990243s) [noise=true , address=45.91.116.14:4189 , piece_id=I66SBU5ZZZOB2RZLT4RHBOBWH3O76Y52BUZ24V57TCVN6SFKTPHQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.607819686 (321h37m26.069912159s) [noise=true , address=45.91.116.7:4010 , piece_id=OG6D4RRIIGRAQWPZR5D2KNJEQMZEZ4NKRNDEMGIN3ILN3G4A2PNA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.608166657 (321h37m26.07025914s) [noise=true , address=77.90.31.16:4733 , piece_id=QRYW55HXATB7KCB2R4APO3XLHXZ37IDHBUPKSDDKYH4ES6TNLFHA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.608429166 (321h37m26.070521639s) [noise=true , address=77.90.31.19:4841 , piece_id=FWKKMSGK6VRG2FZE4WYQXHJJ457Q6G6LSH73CMDFEZI3HDSYZPHQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.609327597 (321h37m26.07142008s) [noise=true , address=145.79.227.4:4187 , piece_id=FAPPF3BE7K2K6NTBYCEOO7OSXYWLPMFHOZPFBRY7GIQA32WFUYQQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.609673586 (321h37m26.071766069s) [noise=true , address=145.79.227.13:4802 , piece_id=77HDEXYCREA2LKC6C2EO5C5WNOG7M6UITBMPDHPNP7FE2VFXL3SA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610149363 (321h37m26.072241846s) [noise=true , address=77.90.31.10:4517 , piece_id=AFEEPAOAMFB6UBI7Q2ZYD23UW6FVLKWWCOS6Q2ASZYOC6IJY4CUA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610443031 (321h37m26.072535514s) [noise=true , address=145.79.227.3:4150 , piece_id=G425E66UIEXC7ALQN7XWF3UAMEVJ2BVYCVHJH4GLNI5DAUKLPCDA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610779452 (321h37m26.072871925s) [noise=true , address=45.91.116.9:4262 , piece_id=SZLNQNID7E4AGEKEKTV7YNWAMHMTEQBRFZAN7EJOJMORA5KOQPHA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.611049776 (321h37m26.073142259s) [noise=true , address=77.90.31.13:4627 , piece_id=2VKP4IMX72TPKWVZMI342SAPQEXJO7EYJNUHM5AE6USW6ZV3MFXQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.611525232 (321h37m26.073617705s) [noise=true , address=145.79.227.11:4731 , piece_id=V7BKZC6KZ3FEWACEIBSTA2JH7ZRLYOV6HYPKZ7OG4C2EWNHOYC4A, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.613139946 (321h37m26.075232429s) [noise=true , address=145.79.227.16:4912 , piece_id=L4TX4F5H2MMAPLFJC6M6S34UAJFKNC5V7WNM6ZQPYPLQ2AGJVDWA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.614580790 (321h37m26.076673273s) [noise=true , address=145.79.227.15:4876 , piece_id=TK6ERRTVD3BT37EXDIRFECXEATGDM4KMOR7PKFMNMO3G7WBN45UQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.615935741 (321h37m26.078028224s) [noise=true , address=77.90.31.12:4590 , piece_id=UTFFG7EQFGLER2TEBXJCFDCKH7PQFH3KPPNJC52BMB5QFQHKHDKQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.617042739 (321h37m26.079135222s) [noise=true , address=145.79.227.12:4767 , piece_id=NV3X2HUD2O4I2ZLAEV7KJXZK4RP2F2FVEQWWKQVE7SBTC43WKXQQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.618171928 (321h37m26.080264401s) [noise=true , address=45.91.116.13:4153 , piece_id=UOR5VXXBAXH5TNJN5QSE6QWG75HWJ27K7YNRJ5QIKRKVOFOBD43Q, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:42.322233711 (321h37m27.784326184s) [upload=283060]: Next returned nil segment Oct 6 23:08:42.322257837 (321h37m27.78435031s) [upload=283060]: Waiting for error group managing segments... Oct 6 23:08:42.596314559 (321h37m28.058407042s) [noise=true , address=145.79.227.15:4876 , piece_id=TK6ERRTVD3BT37EXDIRFECXEATGDM4KMOR7PKFMNMO3G7WBN45UQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.612921512 (321h37m28.075013995s) [noise=true , address=145.79.227.4:4187 , piece_id=FAPPF3BE7K2K6NTBYCEOO7OSXYWLPMFHOZPFBRY7GIQA32WFUYQQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.613258674 (321h37m28.075351157s) [noise=true , address=145.79.227.13:4802 , piece_id=77HDEXYCREA2LKC6C2EO5C5WNOG7M6UITBMPDHPNP7FE2VFXL3SA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.636581324 (321h37m28.098673797s) [noise=true , address=145.79.227.3:4150 , piece_id=G425E66UIEXC7ALQN7XWF3UAMEVJ2BVYCVHJH4GLNI5DAUKLPCDA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.643457217 (321h37m28.1055497s) [noise=true , address=145.79.227.11:4731 , piece_id=V7BKZC6KZ3FEWACEIBSTA2JH7ZRLYOV6HYPKZ7OG4C2EWNHOYC4A, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.643664322 (321h37m28.105756795s) [noise=true , address=145.79.227.12:4767 , piece_id=NV3X2HUD2O4I2ZLAEV7KJXZK4RP2F2FVEQWWKQVE7SBTC43WKXQQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.651495464 (321h37m28.113587937s) [noise=true , address=145.79.227.16:4912 , piece_id=L4TX4F5H2MMAPLFJC6M6S34UAJFKNC5V7WNM6ZQPYPLQ2AGJVDWA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:09:09.649271832 (321h37m55.111364315s) [noise=true , address=77.90.31.19:4841 , piece_id=FWKKMSGK6VRG2FZE4WYQXHJJ457Q6G6LSH73CMDFEZI3HDSYZPHQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 188.166.225.170:52132->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649347195 (321h37m55.111439678s) [noise=true , address=77.90.31.13:4627 , piece_id=2VKP4IMX72TPKWVZMI342SAPQEXJO7EYJNUHM5AE6USW6ZV3MFXQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: rpc: dial tcp 77.90.31.13:4627: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649381470 (321h37m55.111473943s) [noise=true , address=77.90.31.12:4590 , piece_id=UTFFG7EQFGLER2TEBXJCFDCKH7PQFH3KPPNJC52BMB5QFQHKHDKQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 188.166.225.170:43596->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649473486 (321h37m55.111565969s) [noise=true , address=45.91.116.13:4153 , piece_id=UOR5VXXBAXH5TNJN5QSE6QWG75HWJ27K7YNRJ5QIKRKVOFOBD43Q, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 188.166.225.170:46146->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649524713 (321h37m55.111617196s) [noise=true , address=45.91.116.14:4189 , piece_id=I66SBU5ZZZOB2RZLT4RHBOBWH3O76Y52BUZ24V57TCVN6SFKTPHQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:41770->45.91.116.14:4189: use of closed network connection; close tcp 188.166.225.170:41770->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649565671 (321h37m55.111658154s) [noise=true , address=77.90.31.16:4733 , piece_id=QRYW55HXATB7KCB2R4APO3XLHXZ37IDHBUPKSDDKYH4ES6TNLFHA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: read tcp 188.166.225.170:33654->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649671793 (321h37m55.111764276s) [noise=true , address=45.91.116.9:4262 , piece_id=SZLNQNID7E4AGEKEKTV7YNWAMHMTEQBRFZAN7EJOJMORA5KOQPHA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:40868->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649753749 (321h37m55.111846232s) [noise=true , address=45.91.116.7:4010 , piece_id=OG6D4RRIIGRAQWPZR5D2KNJEQMZEZ4NKRNDEMGIN3ILN3G4A2PNA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59266->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649898886 (321h37m55.111991359s) [noise=true , address=77.90.31.10:4517 , piece_id=AFEEPAOAMFB6UBI7Q2ZYD23UW6FVLKWWCOS6Q2ASZYOC6IJY4CUA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:09:09.649942308 (321h37m55.112034791s) [seg_pos={0 0}, upload=283060]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Oct 6 23:09:09.896191619 (321h37m55.358284102s) [upload=283060]: Done uploading segments... Oct 6 23:09:09.896225955 (321h37m55.358318438s) [upload=283060]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 6.554195881s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..................X.....] duration: 6.39091953s FAIL (2025-10-06 23:10:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 21h34m30.318324337s) [upload=283020]: Got next segment. Inline: false Oct 6 23:05:44.856263134 (321h34m30.318355617s) [upload=283020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:44.856268845 (321h34m30.318361328s) [upload=283020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:45.108705808 (321h34m30.570798291s) [seg_pos={0 0}, upload=283020]: Begin upload segment... Oct 6 23:05:45.108853299 (321h34m30.570945772s) [seg_pos={0 0}, upload=283020]: Done begin upload segment. Oct 6 23:05:45.109033753 (321h34m30.571126226s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109109477 (321h34m30.571213192s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109722314 (321h34m30.571814797s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110354308 (321h34m30.572446791s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110950954 (321h34m30.573043437s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.111551888 (321h34m30.573644371s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112154475 (321h34m30.574246958s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112932658 (321h34m30.575025141s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.113602423 (321h34m30.575694896s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.114401965 (321h34m30.576494448s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.116415860 (321h34m30.578508333s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.118906663 (321h34m30.580999136s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.121067647 (321h34m30.58316012s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.129477842 (321h34m30.591570325s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.130084146 (321h34m30.592176629s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.131174373 (321h34m30.593266866s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.133360686 (321h34m30.595453169s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.135350114 (321h34m30.597442597s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:46.574429033 (321h34m32.036521506s) [upload=283020]: Next returned nil segment Oct 6 23:05:46.574445103 (321h34m32.036537576s) [upload=283020]: Waiting for error group managing segments... Oct 6 23:05:46.946795634 (321h34m32.408888107s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.970438174 (321h34m32.432530647s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.971841325 (321h34m32.433933808s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.982490243 (321h34m32.444582726s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.990921268 (321h34m32.453013741s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.993986244 (321h34m32.456078727s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.998097392 (321h34m32.460189875s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:24.533854514 (321h35m9.995946987s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.415951231 (321h35m10.878043704s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.820324670 (321h35m11.282417143s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:27.866791732 (321h35m13.328884215s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:28.427216598 (321h35m13.889309071s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:35.312697524 (321h35m20.774789997s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:44.135928415 (321h35m29.598020888s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1wR3ntox5QJSB1n45AmvQ94C6yzgihoBVYPMjsrgzTtxzhYHy1, address:45.91.116.4:4951): protocol: rpc: dial tcp 45.91.116.4:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136059113 (321h35m29.598151596s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: read tcp 188.166.225.170:41850->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136082528 (321h35m29.598175001s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: rpc: dial tcp 77.90.31.19:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136120630 (321h35m29.598213103s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX, address:45.91.116.6:4004): protocol: rpc: dial tcp 45.91.116.6:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136148143 (321h35m29.598240616s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: read tcp 188.166.225.170:47654->77.90.31.14:4669: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136173562 (321h35m29.598266035s) [seg_pos={0 0}, upload=283020]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 23:06:44.427092462 (321h35m29.889184945s) [upload=283020]: Done uploading segments... Oct 6 23:06:44.427128952 (321h35m29.889221435s) [upload=283020]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.390952923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................XX........X.............] duration: 6.470503786s FAIL (2025-10-06 23:10:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.470535817s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X...........XX.......X..X.] duration: 7.191383788s FAIL (2025-10-06 23:09:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 0]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:08:40.607730747 (321h37m26.06982322s) [seg_pos={0 0}, upload=283060]: Begin upload segment... Oct 6 23:08:40.607788537 (321h37m26.06988101s) [seg_pos={0 0}, upload=283060]: Done begin upload segment. Oct 6 23:08:40.607809947 (321h37m26.06990243s) [noise=true , address=45.91.116.14:4189 , piece_id=I66SBU5ZZZOB2RZLT4RHBOBWH3O76Y52BUZ24V57TCVN6SFKTPHQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.607819686 (321h37m26.069912159s) [noise=true , address=45.91.116.7:4010 , piece_id=OG6D4RRIIGRAQWPZR5D2KNJEQMZEZ4NKRNDEMGIN3ILN3G4A2PNA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.608166657 (321h37m26.07025914s) [noise=true , address=77.90.31.16:4733 , piece_id=QRYW55HXATB7KCB2R4APO3XLHXZ37IDHBUPKSDDKYH4ES6TNLFHA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.608429166 (321h37m26.070521639s) [noise=true , address=77.90.31.19:4841 , piece_id=FWKKMSGK6VRG2FZE4WYQXHJJ457Q6G6LSH73CMDFEZI3HDSYZPHQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.609327597 (321h37m26.07142008s) [noise=true , address=145.79.227.4:4187 , piece_id=FAPPF3BE7K2K6NTBYCEOO7OSXYWLPMFHOZPFBRY7GIQA32WFUYQQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.609673586 (321h37m26.071766069s) [noise=true , address=145.79.227.13:4802 , piece_id=77HDEXYCREA2LKC6C2EO5C5WNOG7M6UITBMPDHPNP7FE2VFXL3SA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610149363 (321h37m26.072241846s) [noise=true , address=77.90.31.10:4517 , piece_id=AFEEPAOAMFB6UBI7Q2ZYD23UW6FVLKWWCOS6Q2ASZYOC6IJY4CUA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610443031 (321h37m26.072535514s) [noise=true , address=145.79.227.3:4150 , piece_id=G425E66UIEXC7ALQN7XWF3UAMEVJ2BVYCVHJH4GLNI5DAUKLPCDA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610779452 (321h37m26.072871925s) [noise=true , address=45.91.116.9:4262 , piece_id=SZLNQNID7E4AGEKEKTV7YNWAMHMTEQBRFZAN7EJOJMORA5KOQPHA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.611049776 (321h37m26.073142259s) [noise=true , address=77.90.31.13:4627 , piece_id=2VKP4IMX72TPKWVZMI342SAPQEXJO7EYJNUHM5AE6USW6ZV3MFXQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.611525232 (321h37m26.073617705s) [noise=true , address=145.79.227.11:4731 , piece_id=V7BKZC6KZ3FEWACEIBSTA2JH7ZRLYOV6HYPKZ7OG4C2EWNHOYC4A, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.613139946 (321h37m26.075232429s) [noise=true , address=145.79.227.16:4912 , piece_id=L4TX4F5H2MMAPLFJC6M6S34UAJFKNC5V7WNM6ZQPYPLQ2AGJVDWA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.614580790 (321h37m26.076673273s) [noise=true , address=145.79.227.15:4876 , piece_id=TK6ERRTVD3BT37EXDIRFECXEATGDM4KMOR7PKFMNMO3G7WBN45UQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.615935741 (321h37m26.078028224s) [noise=true , address=77.90.31.12:4590 , piece_id=UTFFG7EQFGLER2TEBXJCFDCKH7PQFH3KPPNJC52BMB5QFQHKHDKQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.617042739 (321h37m26.079135222s) [noise=true , address=145.79.227.12:4767 , piece_id=NV3X2HUD2O4I2ZLAEV7KJXZK4RP2F2FVEQWWKQVE7SBTC43WKXQQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.618171928 (321h37m26.080264401s) [noise=true , address=45.91.116.13:4153 , piece_id=UOR5VXXBAXH5TNJN5QSE6QWG75HWJ27K7YNRJ5QIKRKVOFOBD43Q, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:42.322233711 (321h37m27.784326184s) [upload=283060]: Next returned nil segment Oct 6 23:08:42.322257837 (321h37m27.78435031s) [upload=283060]: Waiting for error group managing segments... Oct 6 23:08:42.596314559 (321h37m28.058407042s) [noise=true , address=145.79.227.15:4876 , piece_id=TK6ERRTVD3BT37EXDIRFECXEATGDM4KMOR7PKFMNMO3G7WBN45UQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.612921512 (321h37m28.075013995s) [noise=true , address=145.79.227.4:4187 , piece_id=FAPPF3BE7K2K6NTBYCEOO7OSXYWLPMFHOZPFBRY7GIQA32WFUYQQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.613258674 (321h37m28.075351157s) [noise=true , address=145.79.227.13:4802 , piece_id=77HDEXYCREA2LKC6C2EO5C5WNOG7M6UITBMPDHPNP7FE2VFXL3SA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.636581324 (321h37m28.098673797s) [noise=true , address=145.79.227.3:4150 , piece_id=G425E66UIEXC7ALQN7XWF3UAMEVJ2BVYCVHJH4GLNI5DAUKLPCDA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.643457217 (321h37m28.1055497s) [noise=true , address=145.79.227.11:4731 , piece_id=V7BKZC6KZ3FEWACEIBSTA2JH7ZRLYOV6HYPKZ7OG4C2EWNHOYC4A, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.643664322 (321h37m28.105756795s) [noise=true , address=145.79.227.12:4767 , piece_id=NV3X2HUD2O4I2ZLAEV7KJXZK4RP2F2FVEQWWKQVE7SBTC43WKXQQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.651495464 (321h37m28.113587937s) [noise=true , address=145.79.227.16:4912 , piece_id=L4TX4F5H2MMAPLFJC6M6S34UAJFKNC5V7WNM6ZQPYPLQ2AGJVDWA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:09:09.649271832 (321h37m55.111364315s) [noise=true , address=77.90.31.19:4841 , piece_id=FWKKMSGK6VRG2FZE4WYQXHJJ457Q6G6LSH73CMDFEZI3HDSYZPHQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 188.166.225.170:52132->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649347195 (321h37m55.111439678s) [noise=true , address=77.90.31.13:4627 , piece_id=2VKP4IMX72TPKWVZMI342SAPQEXJO7EYJNUHM5AE6USW6ZV3MFXQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: rpc: dial tcp 77.90.31.13:4627: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649381470 (321h37m55.111473943s) [noise=true , address=77.90.31.12:4590 , piece_id=UTFFG7EQFGLER2TEBXJCFDCKH7PQFH3KPPNJC52BMB5QFQHKHDKQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 188.166.225.170:43596->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649473486 (321h37m55.111565969s) [noise=true , address=45.91.116.13:4153 , piece_id=UOR5VXXBAXH5TNJN5QSE6QWG75HWJ27K7YNRJ5QIKRKVOFOBD43Q, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 188.166.225.170:46146->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649524713 (321h37m55.111617196s) [noise=true , address=45.91.116.14:4189 , piece_id=I66SBU5ZZZOB2RZLT4RHBOBWH3O76Y52BUZ24V57TCVN6SFKTPHQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:41770->45.91.116.14:4189: use of closed network connection; close tcp 188.166.225.170:41770->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649565671 (321h37m55.111658154s) [noise=true , address=77.90.31.16:4733 , piece_id=QRYW55HXATB7KCB2R4APO3XLHXZ37IDHBUPKSDDKYH4ES6TNLFHA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: read tcp 188.166.225.170:33654->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649671793 (321h37m55.111764276s) [noise=true , address=45.91.116.9:4262 , piece_id=SZLNQNID7E4AGEKEKTV7YNWAMHMTEQBRFZAN7EJOJMORA5KOQPHA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:40868->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649753749 (321h37m55.111846232s) [noise=true , address=45.91.116.7:4010 , piece_id=OG6D4RRIIGRAQWPZR5D2KNJEQMZEZ4NKRNDEMGIN3ILN3G4A2PNA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59266->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649898886 (321h37m55.111991359s) [noise=true , address=77.90.31.10:4517 , piece_id=AFEEPAOAMFB6UBI7Q2ZYD23UW6FVLKWWCOS6Q2ASZYOC6IJY4CUA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:09:09.649942308 (321h37m55.112034791s) [seg_pos={0 0}, upload=283060]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Oct 6 23:09:09.896191619 (321h37m55.358284102s) [upload=283060]: Done uploading segments... Oct 6 23:09:09.896225955 (321h37m55.358318438s) [upload=283060]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 7.191419408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..................X....] duration: 22.106197841s FAIL (2025-10-06 23:09:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 21h34m30.318324337s) [upload=283020]: Got next segment. Inline: false Oct 6 23:05:44.856263134 (321h34m30.318355617s) [upload=283020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:44.856268845 (321h34m30.318361328s) [upload=283020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:45.108705808 (321h34m30.570798291s) [seg_pos={0 0}, upload=283020]: Begin upload segment... Oct 6 23:05:45.108853299 (321h34m30.570945772s) [seg_pos={0 0}, upload=283020]: Done begin upload segment. Oct 6 23:05:45.109033753 (321h34m30.571126226s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109109477 (321h34m30.571213192s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109722314 (321h34m30.571814797s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110354308 (321h34m30.572446791s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110950954 (321h34m30.573043437s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.111551888 (321h34m30.573644371s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112154475 (321h34m30.574246958s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112932658 (321h34m30.575025141s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.113602423 (321h34m30.575694896s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.114401965 (321h34m30.576494448s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.116415860 (321h34m30.578508333s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.118906663 (321h34m30.580999136s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.121067647 (321h34m30.58316012s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.129477842 (321h34m30.591570325s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.130084146 (321h34m30.592176629s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.131174373 (321h34m30.593266866s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.133360686 (321h34m30.595453169s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.135350114 (321h34m30.597442597s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:46.574429033 (321h34m32.036521506s) [upload=283020]: Next returned nil segment Oct 6 23:05:46.574445103 (321h34m32.036537576s) [upload=283020]: Waiting for error group managing segments... Oct 6 23:05:46.946795634 (321h34m32.408888107s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.970438174 (321h34m32.432530647s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.971841325 (321h34m32.433933808s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.982490243 (321h34m32.444582726s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.990921268 (321h34m32.453013741s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.993986244 (321h34m32.456078727s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.998097392 (321h34m32.460189875s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:24.533854514 (321h35m9.995946987s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.415951231 (321h35m10.878043704s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.820324670 (321h35m11.282417143s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:27.866791732 (321h35m13.328884215s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:28.427216598 (321h35m13.889309071s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:35.312697524 (321h35m20.774789997s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:44.135928415 (321h35m29.598020888s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1wR3ntox5QJSB1n45AmvQ94C6yzgihoBVYPMjsrgzTtxzhYHy1, address:45.91.116.4:4951): protocol: rpc: dial tcp 45.91.116.4:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136059113 (321h35m29.598151596s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: read tcp 188.166.225.170:41850->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136082528 (321h35m29.598175001s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: rpc: dial tcp 77.90.31.19:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136120630 (321h35m29.598213103s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX, address:45.91.116.6:4004): protocol: rpc: dial tcp 45.91.116.6:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136148143 (321h35m29.598240616s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: read tcp 188.166.225.170:47654->77.90.31.14:4669: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136173562 (321h35m29.598266035s) [seg_pos={0 0}, upload=283020]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 23:06:44.427092462 (321h35m29.889184945s) [upload=283020]: Done uploading segments... Oct 6 23:06:44.427128952 (321h35m29.889221435s) [upload=283020]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.106228199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XX........X............] duration: 6.154272533s FAIL (2025-10-06 23:09:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.154297942s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................X...........XX.......X..X] duration: 30.249109657s FAIL (2025-10-06 23:09:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 0]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:08:40.607730747 (321h37m26.06982322s) [seg_pos={0 0}, upload=283060]: Begin upload segment... Oct 6 23:08:40.607788537 (321h37m26.06988101s) [seg_pos={0 0}, upload=283060]: Done begin upload segment. Oct 6 23:08:40.607809947 (321h37m26.06990243s) [noise=true , address=45.91.116.14:4189 , piece_id=I66SBU5ZZZOB2RZLT4RHBOBWH3O76Y52BUZ24V57TCVN6SFKTPHQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.607819686 (321h37m26.069912159s) [noise=true , address=45.91.116.7:4010 , piece_id=OG6D4RRIIGRAQWPZR5D2KNJEQMZEZ4NKRNDEMGIN3ILN3G4A2PNA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.608166657 (321h37m26.07025914s) [noise=true , address=77.90.31.16:4733 , piece_id=QRYW55HXATB7KCB2R4APO3XLHXZ37IDHBUPKSDDKYH4ES6TNLFHA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.608429166 (321h37m26.070521639s) [noise=true , address=77.90.31.19:4841 , piece_id=FWKKMSGK6VRG2FZE4WYQXHJJ457Q6G6LSH73CMDFEZI3HDSYZPHQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.609327597 (321h37m26.07142008s) [noise=true , address=145.79.227.4:4187 , piece_id=FAPPF3BE7K2K6NTBYCEOO7OSXYWLPMFHOZPFBRY7GIQA32WFUYQQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.609673586 (321h37m26.071766069s) [noise=true , address=145.79.227.13:4802 , piece_id=77HDEXYCREA2LKC6C2EO5C5WNOG7M6UITBMPDHPNP7FE2VFXL3SA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610149363 (321h37m26.072241846s) [noise=true , address=77.90.31.10:4517 , piece_id=AFEEPAOAMFB6UBI7Q2ZYD23UW6FVLKWWCOS6Q2ASZYOC6IJY4CUA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610443031 (321h37m26.072535514s) [noise=true , address=145.79.227.3:4150 , piece_id=G425E66UIEXC7ALQN7XWF3UAMEVJ2BVYCVHJH4GLNI5DAUKLPCDA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.610779452 (321h37m26.072871925s) [noise=true , address=45.91.116.9:4262 , piece_id=SZLNQNID7E4AGEKEKTV7YNWAMHMTEQBRFZAN7EJOJMORA5KOQPHA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.611049776 (321h37m26.073142259s) [noise=true , address=77.90.31.13:4627 , piece_id=2VKP4IMX72TPKWVZMI342SAPQEXJO7EYJNUHM5AE6USW6ZV3MFXQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.611525232 (321h37m26.073617705s) [noise=true , address=145.79.227.11:4731 , piece_id=V7BKZC6KZ3FEWACEIBSTA2JH7ZRLYOV6HYPKZ7OG4C2EWNHOYC4A, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.613139946 (321h37m26.075232429s) [noise=true , address=145.79.227.16:4912 , piece_id=L4TX4F5H2MMAPLFJC6M6S34UAJFKNC5V7WNM6ZQPYPLQ2AGJVDWA, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.614580790 (321h37m26.076673273s) [noise=true , address=145.79.227.15:4876 , piece_id=TK6ERRTVD3BT37EXDIRFECXEATGDM4KMOR7PKFMNMO3G7WBN45UQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.615935741 (321h37m26.078028224s) [noise=true , address=77.90.31.12:4590 , piece_id=UTFFG7EQFGLER2TEBXJCFDCKH7PQFH3KPPNJC52BMB5QFQHKHDKQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.617042739 (321h37m26.079135222s) [noise=true , address=145.79.227.12:4767 , piece_id=NV3X2HUD2O4I2ZLAEV7KJXZK4RP2F2FVEQWWKQVE7SBTC43WKXQQ, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:40.618171928 (321h37m26.080264401s) [noise=true , address=45.91.116.13:4153 , piece_id=UOR5VXXBAXH5TNJN5QSE6QWG75HWJ27K7YNRJ5QIKRKVOFOBD43Q, seg_pos={0 0}, upload=283060]: Uploading piece... Oct 6 23:08:42.322233711 (321h37m27.784326184s) [upload=283060]: Next returned nil segment Oct 6 23:08:42.322257837 (321h37m27.78435031s) [upload=283060]: Waiting for error group managing segments... Oct 6 23:08:42.596314559 (321h37m28.058407042s) [noise=true , address=145.79.227.15:4876 , piece_id=TK6ERRTVD3BT37EXDIRFECXEATGDM4KMOR7PKFMNMO3G7WBN45UQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.612921512 (321h37m28.075013995s) [noise=true , address=145.79.227.4:4187 , piece_id=FAPPF3BE7K2K6NTBYCEOO7OSXYWLPMFHOZPFBRY7GIQA32WFUYQQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.613258674 (321h37m28.075351157s) [noise=true , address=145.79.227.13:4802 , piece_id=77HDEXYCREA2LKC6C2EO5C5WNOG7M6UITBMPDHPNP7FE2VFXL3SA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.636581324 (321h37m28.098673797s) [noise=true , address=145.79.227.3:4150 , piece_id=G425E66UIEXC7ALQN7XWF3UAMEVJ2BVYCVHJH4GLNI5DAUKLPCDA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.643457217 (321h37m28.1055497s) [noise=true , address=145.79.227.11:4731 , piece_id=V7BKZC6KZ3FEWACEIBSTA2JH7ZRLYOV6HYPKZ7OG4C2EWNHOYC4A, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.643664322 (321h37m28.105756795s) [noise=true , address=145.79.227.12:4767 , piece_id=NV3X2HUD2O4I2ZLAEV7KJXZK4RP2F2FVEQWWKQVE7SBTC43WKXQQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:08:42.651495464 (321h37m28.113587937s) [noise=true , address=145.79.227.16:4912 , piece_id=L4TX4F5H2MMAPLFJC6M6S34UAJFKNC5V7WNM6ZQPYPLQ2AGJVDWA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: <nil> Oct 6 23:09:09.649271832 (321h37m55.111364315s) [noise=true , address=77.90.31.19:4841 , piece_id=FWKKMSGK6VRG2FZE4WYQXHJJ457Q6G6LSH73CMDFEZI3HDSYZPHQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 188.166.225.170:52132->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649347195 (321h37m55.111439678s) [noise=true , address=77.90.31.13:4627 , piece_id=2VKP4IMX72TPKWVZMI342SAPQEXJO7EYJNUHM5AE6USW6ZV3MFXQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: rpc: dial tcp 77.90.31.13:4627: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649381470 (321h37m55.111473943s) [noise=true , address=77.90.31.12:4590 , piece_id=UTFFG7EQFGLER2TEBXJCFDCKH7PQFH3KPPNJC52BMB5QFQHKHDKQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 188.166.225.170:43596->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649473486 (321h37m55.111565969s) [noise=true , address=45.91.116.13:4153 , piece_id=UOR5VXXBAXH5TNJN5QSE6QWG75HWJ27K7YNRJ5QIKRKVOFOBD43Q, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 188.166.225.170:46146->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649524713 (321h37m55.111617196s) [noise=true , address=45.91.116.14:4189 , piece_id=I66SBU5ZZZOB2RZLT4RHBOBWH3O76Y52BUZ24V57TCVN6SFKTPHQ, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:41770->45.91.116.14:4189: use of closed network connection; close tcp 188.166.225.170:41770->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649565671 (321h37m55.111658154s) [noise=true , address=77.90.31.16:4733 , piece_id=QRYW55HXATB7KCB2R4APO3XLHXZ37IDHBUPKSDDKYH4ES6TNLFHA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: read tcp 188.166.225.170:33654->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649671793 (321h37m55.111764276s) [noise=true , address=45.91.116.9:4262 , piece_id=SZLNQNID7E4AGEKEKTV7YNWAMHMTEQBRFZAN7EJOJMORA5KOQPHA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:40868->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649753749 (321h37m55.111846232s) [noise=true , address=45.91.116.7:4010 , piece_id=OG6D4RRIIGRAQWPZR5D2KNJEQMZEZ4NKRNDEMGIN3ILN3G4A2PNA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59266->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:09:09.649898886 (321h37m55.111991359s) [noise=true , address=77.90.31.10:4517 , piece_id=AFEEPAOAMFB6UBI7Q2ZYD23UW6FVLKWWCOS6Q2ASZYOC6IJY4CUA, seg_pos={0 0}, upload=283060]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:09:09.649942308 (321h37m55.112034791s) [seg_pos={0 0}, upload=283060]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Oct 6 23:09:09.896191619 (321h37m55.358284102s) [upload=283060]: Done uploading segments... Oct 6 23:09:09.896225955 (321h37m55.358318438s) [upload=283060]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 30.249136268s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................XX........X...........] duration: 6.505543197s FAIL (2025-10-06 23:08:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.505563176s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..................X...] duration: 6.161247528s FAIL (2025-10-06 23:08:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 21h34m30.318324337s) [upload=283020]: Got next segment. Inline: false Oct 6 23:05:44.856263134 (321h34m30.318355617s) [upload=283020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:44.856268845 (321h34m30.318361328s) [upload=283020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:45.108705808 (321h34m30.570798291s) [seg_pos={0 0}, upload=283020]: Begin upload segment... Oct 6 23:05:45.108853299 (321h34m30.570945772s) [seg_pos={0 0}, upload=283020]: Done begin upload segment. Oct 6 23:05:45.109033753 (321h34m30.571126226s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109109477 (321h34m30.571213192s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109722314 (321h34m30.571814797s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110354308 (321h34m30.572446791s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110950954 (321h34m30.573043437s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.111551888 (321h34m30.573644371s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112154475 (321h34m30.574246958s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112932658 (321h34m30.575025141s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.113602423 (321h34m30.575694896s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.114401965 (321h34m30.576494448s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.116415860 (321h34m30.578508333s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.118906663 (321h34m30.580999136s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.121067647 (321h34m30.58316012s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.129477842 (321h34m30.591570325s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.130084146 (321h34m30.592176629s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.131174373 (321h34m30.593266866s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.133360686 (321h34m30.595453169s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.135350114 (321h34m30.597442597s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:46.574429033 (321h34m32.036521506s) [upload=283020]: Next returned nil segment Oct 6 23:05:46.574445103 (321h34m32.036537576s) [upload=283020]: Waiting for error group managing segments... Oct 6 23:05:46.946795634 (321h34m32.408888107s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.970438174 (321h34m32.432530647s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.971841325 (321h34m32.433933808s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.982490243 (321h34m32.444582726s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.990921268 (321h34m32.453013741s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.993986244 (321h34m32.456078727s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.998097392 (321h34m32.460189875s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:24.533854514 (321h35m9.995946987s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.415951231 (321h35m10.878043704s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.820324670 (321h35m11.282417143s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:27.866791732 (321h35m13.328884215s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:28.427216598 (321h35m13.889309071s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:35.312697524 (321h35m20.774789997s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:44.135928415 (321h35m29.598020888s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1wR3ntox5QJSB1n45AmvQ94C6yzgihoBVYPMjsrgzTtxzhYHy1, address:45.91.116.4:4951): protocol: rpc: dial tcp 45.91.116.4:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136059113 (321h35m29.598151596s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: read tcp 188.166.225.170:41850->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136082528 (321h35m29.598175001s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: rpc: dial tcp 77.90.31.19:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136120630 (321h35m29.598213103s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX, address:45.91.116.6:4004): protocol: rpc: dial tcp 45.91.116.6:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136148143 (321h35m29.598240616s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: read tcp 188.166.225.170:47654->77.90.31.14:4669: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136173562 (321h35m29.598266035s) [seg_pos={0 0}, upload=283020]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 23:06:44.427092462 (321h35m29.889184945s) [upload=283020]: Done uploading segments... Oct 6 23:06:44.427128952 (321h35m29.889221435s) [upload=283020]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.161291261s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XX........X..........] duration: 6.426679385s FAIL (2025-10-06 23:08:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.426707619s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X...........XX.......X..] duration: 6.710455303s FAIL (2025-10-06 23:07:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:05:56.416568715 (321h34m41.878661188s) [upload=283023]: Starting upload Oct 6 23:05:56.416662404 (321h34m41.878754877s) [upload=283023]: Done starting upload Oct 6 23:05:56.417037158 (321h34m41.879129641s) [upload=283023]: Uploading segments... Oct 6 23:05:56.417096550 (321h34m41.879189023s) [upload=283023]: Got next segment. Inline: false Oct 6 23:05:56.417129693 (321h34m41.879222166s) [upload=283023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:56.417134293 (321h34m41.879226776s) [upload=283023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:56.670774287 (321h34m42.13286676s) [seg_pos={0 0}, upload=283023]: Begin upload segment... Oct 6 23:05:56.670846254 (321h34m42.132938727s) [seg_pos={0 0}, upload=283023]: Done begin upload segment. Oct 6 23:05:56.670877513 (321h34m42.132969987s) [noise=true , address=77.90.31.10:4517 , piece_id=TZAUEVDKXMXPU2G6QV5CNC2UTACQS4BY33EXIKHQEXAFEU3GI2CA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.671620058 (321h34m42.133712541s) [noise=true , address=45.91.116.8:4048 , piece_id=DNWNNHTKCKDN2JD4TJHI5DNUCYCDKQADSRMOYDLPLRCP6FYC4KLQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.671940328 (321h34m42.134032811s) [noise=true , address=145.79.227.3:4151 , piece_id=3PJVRPTKGC6HZRGNYX4XQKNF77XYVGUORMN7MZVDPPXOEA6OPRIQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.672214861 (321h34m42.134307334s) [noise=true , address=45.91.116.14:4190 , piece_id=4FWYOHPF2YGKVBUARUUKIC677B75P4MKCMBVDJAX42WBRCS4KAZQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.672588331 (321h34m42.134680814s) [noise=true , address=145.79.227.11:4732 , piece_id=N27D6TT34JUUBKZG4EEW6BKCLJ7SYQXQTHPYGPRYCCONCPAFJBZA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673102771 (321h34m42.135195254s) [noise=true , address=45.91.116.12:4225 , piece_id=VO2YOOXDFK5RRIZKTD5ML5G73KONYZBH766OGHF3L66TEM2QA2LQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673502983 (321h34m42.135595476s) [noise=true , address=77.90.31.13:4628 , piece_id=LVFG563ROTB7NVWKM4ICZ6WRVAM3SAMF3SDE2D72FDR52PPF2UHA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673964462 (321h34m42.136056945s) [noise=true , address=145.79.227.2:4114 , piece_id=FTD3NMHADI245UVXUPNMGZEB2M3VJYKSBIT3MNWHG4IY5EPPKGQA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.675410636 (321h34m42.137503109s) [noise=true , address=45.91.116.9:4263 , piece_id=YYVJROMPN2FZU3S3EHQT74DU24JJ6YZU3FPT6I5JUPNOYVUXW5OA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.676852271 (321h34m42.138944754s) [noise=true , address=45.91.116.5:4298 , piece_id=4OBCY25BUZOQJJMGI6S4LXWRPUTG7FJTJYSYICSPIWUG2IJNBVTA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.680370049 (321h34m42.142462522s) [noise=true , address=145.79.227.14:4837 , piece_id=4VAZSD6ZX4HRF3HETNZR56ZFHSP6ZOOYXRBXQTEAYGFQNXJOFAUQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.682234680 (321h34m42.144327163s) [noise=true , address=145.79.227.15:4874 , piece_id=POV5RP5RPFLI4FILLPQ3TWW3O5C76U6ISO3ITILOOPSQFJDSPJIA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.683379109 (321h34m42.145471582s) [noise=true , address=145.79.227.5:4221 , piece_id=IK6CBA5POZ3TICDOPQSD73IWH3QZQRTECCTYG6KQC2IYY7AV7U4Q, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.683555826 (321h34m42.145648309s) [noise=true , address=145.79.227.17:4970 , piece_id=DWBWSXLIYNTH6PHNZ5JJWU5XJAEX3XXVHPZVTTTN6VUUZUZWJ54Q, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.685705879 (321h34m42.147798362s) [noise=true , address=145.79.227.4:4187 , piece_id=SPMGKOXWH2OOOSTR7UHVFSG64BO3IKFDGYFOQBJA2JJ65JLBH2UA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.686479692 (321h34m42.148572167s) [noise=true , address=145.79.227.16:4911 , piece_id=JYL3UV2P3EEYQFZB5M2EKBWE4SUZOLPEFKGQ6NHDHGIQLESBYAIQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:58.364067654 (321h34m43.826160137s) [upload=283023]: Next returned nil segment Oct 6 23:05:58.364086951 (321h34m43.826179434s) [upload=283023]: Waiting for error group managing segments... Oct 6 23:05:58.651101546 (321h34m44.11319404s) [noise=true , address=145.79.227.5:4221 , piece_id=IK6CBA5POZ3TICDOPQSD73IWH3QZQRTECCTYG6KQC2IYY7AV7U4Q, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.662711564 (321h34m44.124804047s) [noise=true , address=145.79.227.4:4187 , piece_id=SPMGKOXWH2OOOSTR7UHVFSG64BO3IKFDGYFOQBJA2JJ65JLBH2UA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.679699713 (321h34m44.141792196s) [noise=true , address=145.79.227.11:4732 , piece_id=N27D6TT34JUUBKZG4EEW6BKCLJ7SYQXQTHPYGPRYCCONCPAFJBZA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.679802989 (321h34m44.141895472s) [noise=true , address=145.79.227.2:4114 , piece_id=FTD3NMHADI245UVXUPNMGZEB2M3VJYKSBIT3MNWHG4IY5EPPKGQA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.691271527 (321h34m44.153364s) [noise=true , address=145.79.227.16:4911 , piece_id=JYL3UV2P3EEYQFZB5M2EKBWE4SUZOLPEFKGQ6NHDHGIQLESBYAIQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.692417900 (321h34m44.154510383s) [noise=true , address=145.79.227.17:4970 , piece_id=DWBWSXLIYNTH6PHNZ5JJWU5XJAEX3XXVHPZVTTTN6VUUZUZWJ54Q, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.694061870 (321h34m44.156154353s) [noise=true , address=145.79.227.15:4874 , piece_id=POV5RP5RPFLI4FILLPQ3TWW3O5C76U6ISO3ITILOOPSQFJDSPJIA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.707610320 (321h34m44.169702803s) [noise=true , address=145.79.227.3:4151 , piece_id=3PJVRPTKGC6HZRGNYX4XQKNF77XYVGUORMN7MZVDPPXOEA6OPRIQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.735344851 (321h34m44.197437324s) [noise=true , address=145.79.227.14:4837 , piece_id=4VAZSD6ZX4HRF3HETNZR56ZFHSP6ZOOYXRBXQTEAYGFQNXJOFAUQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:06:25.703533147 (321h35m11.16562563s) [noise=true , address=77.90.31.10:4517 , piece_id=TZAUEVDKXMXPU2G6QV5CNC2UTACQS4BY33EXIKHQEXAFEU3GI2CA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:36946->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703630603 (321h35m11.165723086s) [noise=true , address=45.91.116.8:4048 , piece_id=DNWNNHTKCKDN2JD4TJHI5DNUCYCDKQADSRMOYDLPLRCP6FYC4KLQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:48728->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703721517 (321h35m11.16581399s) [noise=true , address=45.91.116.14:4190 , piece_id=4FWYOHPF2YGKVBUARUUKIC677B75P4MKCMBVDJAX42WBRCS4KAZQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52264->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703801208 (321h35m11.165893681s) [noise=true , address=45.91.116.9:4263 , piece_id=YYVJROMPN2FZU3S3EHQT74DU24JJ6YZU3FPT6I5JUPNOYVUXW5OA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:34600->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703948989 (321h35m11.166041462s) [noise=true , address=45.91.116.5:4298 , piece_id=4OBCY25BUZOQJJMGI6S4LXWRPUTG7FJTJYSYICSPIWUG2IJNBVTA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704020986 (321h35m11.166113459s) [noise=true , address=77.90.31.13:4628 , piece_id=LVFG563ROTB7NVWKM4ICZ6WRVAM3SAMF3SDE2D72FDR52PPF2UHA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704125335 (321h35m11.166217818s) [noise=true , address=45.91.116.12:4225 , piece_id=VO2YOOXDFK5RRIZKTD5ML5G73KONYZBH766OGHF3L66TEM2QA2LQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704149722 (321h35m11.166242205s) [seg_pos={0 0}, upload=283023]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:06:25.968228364 (321h35m11.430320837s) [upload=283023]: Done uploading segments... Oct 6 23:06:25.968256157 (321h35m11.43034863s) [upload=283023]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.710486572s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................XX........X.........] duration: 6.318312519s FAIL (2025-10-06 23:07:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.318337237s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X...........XX.......X.] duration: 6.71561511s FAIL (2025-10-06 23:07:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:05:56.416568715 (321h34m41.878661188s) [upload=283023]: Starting upload Oct 6 23:05:56.416662404 (321h34m41.878754877s) [upload=283023]: Done starting upload Oct 6 23:05:56.417037158 (321h34m41.879129641s) [upload=283023]: Uploading segments... Oct 6 23:05:56.417096550 (321h34m41.879189023s) [upload=283023]: Got next segment. Inline: false Oct 6 23:05:56.417129693 (321h34m41.879222166s) [upload=283023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:56.417134293 (321h34m41.879226776s) [upload=283023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:56.670774287 (321h34m42.13286676s) [seg_pos={0 0}, upload=283023]: Begin upload segment... Oct 6 23:05:56.670846254 (321h34m42.132938727s) [seg_pos={0 0}, upload=283023]: Done begin upload segment. Oct 6 23:05:56.670877513 (321h34m42.132969987s) [noise=true , address=77.90.31.10:4517 , piece_id=TZAUEVDKXMXPU2G6QV5CNC2UTACQS4BY33EXIKHQEXAFEU3GI2CA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.671620058 (321h34m42.133712541s) [noise=true , address=45.91.116.8:4048 , piece_id=DNWNNHTKCKDN2JD4TJHI5DNUCYCDKQADSRMOYDLPLRCP6FYC4KLQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.671940328 (321h34m42.134032811s) [noise=true , address=145.79.227.3:4151 , piece_id=3PJVRPTKGC6HZRGNYX4XQKNF77XYVGUORMN7MZVDPPXOEA6OPRIQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.672214861 (321h34m42.134307334s) [noise=true , address=45.91.116.14:4190 , piece_id=4FWYOHPF2YGKVBUARUUKIC677B75P4MKCMBVDJAX42WBRCS4KAZQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.672588331 (321h34m42.134680814s) [noise=true , address=145.79.227.11:4732 , piece_id=N27D6TT34JUUBKZG4EEW6BKCLJ7SYQXQTHPYGPRYCCONCPAFJBZA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673102771 (321h34m42.135195254s) [noise=true , address=45.91.116.12:4225 , piece_id=VO2YOOXDFK5RRIZKTD5ML5G73KONYZBH766OGHF3L66TEM2QA2LQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673502983 (321h34m42.135595476s) [noise=true , address=77.90.31.13:4628 , piece_id=LVFG563ROTB7NVWKM4ICZ6WRVAM3SAMF3SDE2D72FDR52PPF2UHA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673964462 (321h34m42.136056945s) [noise=true , address=145.79.227.2:4114 , piece_id=FTD3NMHADI245UVXUPNMGZEB2M3VJYKSBIT3MNWHG4IY5EPPKGQA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.675410636 (321h34m42.137503109s) [noise=true , address=45.91.116.9:4263 , piece_id=YYVJROMPN2FZU3S3EHQT74DU24JJ6YZU3FPT6I5JUPNOYVUXW5OA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.676852271 (321h34m42.138944754s) [noise=true , address=45.91.116.5:4298 , piece_id=4OBCY25BUZOQJJMGI6S4LXWRPUTG7FJTJYSYICSPIWUG2IJNBVTA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.680370049 (321h34m42.142462522s) [noise=true , address=145.79.227.14:4837 , piece_id=4VAZSD6ZX4HRF3HETNZR56ZFHSP6ZOOYXRBXQTEAYGFQNXJOFAUQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.682234680 (321h34m42.144327163s) [noise=true , address=145.79.227.15:4874 , piece_id=POV5RP5RPFLI4FILLPQ3TWW3O5C76U6ISO3ITILOOPSQFJDSPJIA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.683379109 (321h34m42.145471582s) [noise=true , address=145.79.227.5:4221 , piece_id=IK6CBA5POZ3TICDOPQSD73IWH3QZQRTECCTYG6KQC2IYY7AV7U4Q, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.683555826 (321h34m42.145648309s) [noise=true , address=145.79.227.17:4970 , piece_id=DWBWSXLIYNTH6PHNZ5JJWU5XJAEX3XXVHPZVTTTN6VUUZUZWJ54Q, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.685705879 (321h34m42.147798362s) [noise=true , address=145.79.227.4:4187 , piece_id=SPMGKOXWH2OOOSTR7UHVFSG64BO3IKFDGYFOQBJA2JJ65JLBH2UA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.686479692 (321h34m42.148572167s) [noise=true , address=145.79.227.16:4911 , piece_id=JYL3UV2P3EEYQFZB5M2EKBWE4SUZOLPEFKGQ6NHDHGIQLESBYAIQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:58.364067654 (321h34m43.826160137s) [upload=283023]: Next returned nil segment Oct 6 23:05:58.364086951 (321h34m43.826179434s) [upload=283023]: Waiting for error group managing segments... Oct 6 23:05:58.651101546 (321h34m44.11319404s) [noise=true , address=145.79.227.5:4221 , piece_id=IK6CBA5POZ3TICDOPQSD73IWH3QZQRTECCTYG6KQC2IYY7AV7U4Q, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.662711564 (321h34m44.124804047s) [noise=true , address=145.79.227.4:4187 , piece_id=SPMGKOXWH2OOOSTR7UHVFSG64BO3IKFDGYFOQBJA2JJ65JLBH2UA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.679699713 (321h34m44.141792196s) [noise=true , address=145.79.227.11:4732 , piece_id=N27D6TT34JUUBKZG4EEW6BKCLJ7SYQXQTHPYGPRYCCONCPAFJBZA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.679802989 (321h34m44.141895472s) [noise=true , address=145.79.227.2:4114 , piece_id=FTD3NMHADI245UVXUPNMGZEB2M3VJYKSBIT3MNWHG4IY5EPPKGQA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.691271527 (321h34m44.153364s) [noise=true , address=145.79.227.16:4911 , piece_id=JYL3UV2P3EEYQFZB5M2EKBWE4SUZOLPEFKGQ6NHDHGIQLESBYAIQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.692417900 (321h34m44.154510383s) [noise=true , address=145.79.227.17:4970 , piece_id=DWBWSXLIYNTH6PHNZ5JJWU5XJAEX3XXVHPZVTTTN6VUUZUZWJ54Q, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.694061870 (321h34m44.156154353s) [noise=true , address=145.79.227.15:4874 , piece_id=POV5RP5RPFLI4FILLPQ3TWW3O5C76U6ISO3ITILOOPSQFJDSPJIA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.707610320 (321h34m44.169702803s) [noise=true , address=145.79.227.3:4151 , piece_id=3PJVRPTKGC6HZRGNYX4XQKNF77XYVGUORMN7MZVDPPXOEA6OPRIQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.735344851 (321h34m44.197437324s) [noise=true , address=145.79.227.14:4837 , piece_id=4VAZSD6ZX4HRF3HETNZR56ZFHSP6ZOOYXRBXQTEAYGFQNXJOFAUQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:06:25.703533147 (321h35m11.16562563s) [noise=true , address=77.90.31.10:4517 , piece_id=TZAUEVDKXMXPU2G6QV5CNC2UTACQS4BY33EXIKHQEXAFEU3GI2CA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:36946->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703630603 (321h35m11.165723086s) [noise=true , address=45.91.116.8:4048 , piece_id=DNWNNHTKCKDN2JD4TJHI5DNUCYCDKQADSRMOYDLPLRCP6FYC4KLQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:48728->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703721517 (321h35m11.16581399s) [noise=true , address=45.91.116.14:4190 , piece_id=4FWYOHPF2YGKVBUARUUKIC677B75P4MKCMBVDJAX42WBRCS4KAZQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52264->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703801208 (321h35m11.165893681s) [noise=true , address=45.91.116.9:4263 , piece_id=YYVJROMPN2FZU3S3EHQT74DU24JJ6YZU3FPT6I5JUPNOYVUXW5OA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:34600->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703948989 (321h35m11.166041462s) [noise=true , address=45.91.116.5:4298 , piece_id=4OBCY25BUZOQJJMGI6S4LXWRPUTG7FJTJYSYICSPIWUG2IJNBVTA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704020986 (321h35m11.166113459s) [noise=true , address=77.90.31.13:4628 , piece_id=LVFG563ROTB7NVWKM4ICZ6WRVAM3SAMF3SDE2D72FDR52PPF2UHA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704125335 (321h35m11.166217818s) [noise=true , address=45.91.116.12:4225 , piece_id=VO2YOOXDFK5RRIZKTD5ML5G73KONYZBH766OGHF3L66TEM2QA2LQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704149722 (321h35m11.166242205s) [seg_pos={0 0}, upload=283023]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:06:25.968228364 (321h35m11.430320837s) [upload=283023]: Done uploading segments... Oct 6 23:06:25.968256157 (321h35m11.43034863s) [upload=283023]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.715645929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..................X..] duration: 6.530564611s FAIL (2025-10-06 23:07:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 21h34m30.318324337s) [upload=283020]: Got next segment. Inline: false Oct 6 23:05:44.856263134 (321h34m30.318355617s) [upload=283020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:44.856268845 (321h34m30.318361328s) [upload=283020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:45.108705808 (321h34m30.570798291s) [seg_pos={0 0}, upload=283020]: Begin upload segment... Oct 6 23:05:45.108853299 (321h34m30.570945772s) [seg_pos={0 0}, upload=283020]: Done begin upload segment. Oct 6 23:05:45.109033753 (321h34m30.571126226s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109109477 (321h34m30.571213192s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109722314 (321h34m30.571814797s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110354308 (321h34m30.572446791s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110950954 (321h34m30.573043437s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.111551888 (321h34m30.573644371s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112154475 (321h34m30.574246958s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112932658 (321h34m30.575025141s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.113602423 (321h34m30.575694896s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.114401965 (321h34m30.576494448s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.116415860 (321h34m30.578508333s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.118906663 (321h34m30.580999136s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.121067647 (321h34m30.58316012s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.129477842 (321h34m30.591570325s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.130084146 (321h34m30.592176629s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.131174373 (321h34m30.593266866s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.133360686 (321h34m30.595453169s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.135350114 (321h34m30.597442597s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:46.574429033 (321h34m32.036521506s) [upload=283020]: Next returned nil segment Oct 6 23:05:46.574445103 (321h34m32.036537576s) [upload=283020]: Waiting for error group managing segments... Oct 6 23:05:46.946795634 (321h34m32.408888107s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.970438174 (321h34m32.432530647s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.971841325 (321h34m32.433933808s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.982490243 (321h34m32.444582726s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.990921268 (321h34m32.453013741s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.993986244 (321h34m32.456078727s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.998097392 (321h34m32.460189875s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:24.533854514 (321h35m9.995946987s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.415951231 (321h35m10.878043704s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.820324670 (321h35m11.282417143s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:27.866791732 (321h35m13.328884215s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:28.427216598 (321h35m13.889309071s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:35.312697524 (321h35m20.774789997s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:44.135928415 (321h35m29.598020888s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1wR3ntox5QJSB1n45AmvQ94C6yzgihoBVYPMjsrgzTtxzhYHy1, address:45.91.116.4:4951): protocol: rpc: dial tcp 45.91.116.4:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136059113 (321h35m29.598151596s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: read tcp 188.166.225.170:41850->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136082528 (321h35m29.598175001s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: rpc: dial tcp 77.90.31.19:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136120630 (321h35m29.598213103s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX, address:45.91.116.6:4004): protocol: rpc: dial tcp 45.91.116.6:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136148143 (321h35m29.598240616s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: read tcp 188.166.225.170:47654->77.90.31.14:4669: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136173562 (321h35m29.598266035s) [seg_pos={0 0}, upload=283020]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 23:06:44.427092462 (321h35m29.889184945s) [upload=283020]: Done uploading segments... Oct 6 23:06:44.427128952 (321h35m29.889221435s) [upload=283020]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.530591374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..................X.] duration: 6.595610081s FAIL (2025-10-06 23:06:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 21h34m30.318324337s) [upload=283020]: Got next segment. Inline: false Oct 6 23:05:44.856263134 (321h34m30.318355617s) [upload=283020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:44.856268845 (321h34m30.318361328s) [upload=283020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:45.108705808 (321h34m30.570798291s) [seg_pos={0 0}, upload=283020]: Begin upload segment... Oct 6 23:05:45.108853299 (321h34m30.570945772s) [seg_pos={0 0}, upload=283020]: Done begin upload segment. Oct 6 23:05:45.109033753 (321h34m30.571126226s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109109477 (321h34m30.571213192s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109722314 (321h34m30.571814797s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110354308 (321h34m30.572446791s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110950954 (321h34m30.573043437s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.111551888 (321h34m30.573644371s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112154475 (321h34m30.574246958s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112932658 (321h34m30.575025141s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.113602423 (321h34m30.575694896s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.114401965 (321h34m30.576494448s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.116415860 (321h34m30.578508333s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.118906663 (321h34m30.580999136s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.121067647 (321h34m30.58316012s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.129477842 (321h34m30.591570325s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.130084146 (321h34m30.592176629s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.131174373 (321h34m30.593266866s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.133360686 (321h34m30.595453169s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.135350114 (321h34m30.597442597s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:46.574429033 (321h34m32.036521506s) [upload=283020]: Next returned nil segment Oct 6 23:05:46.574445103 (321h34m32.036537576s) [upload=283020]: Waiting for error group managing segments... Oct 6 23:05:46.946795634 (321h34m32.408888107s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.970438174 (321h34m32.432530647s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.971841325 (321h34m32.433933808s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.982490243 (321h34m32.444582726s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.990921268 (321h34m32.453013741s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.993986244 (321h34m32.456078727s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.998097392 (321h34m32.460189875s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:24.533854514 (321h35m9.995946987s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.415951231 (321h35m10.878043704s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.820324670 (321h35m11.282417143s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:27.866791732 (321h35m13.328884215s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:28.427216598 (321h35m13.889309071s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:35.312697524 (321h35m20.774789997s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:44.135928415 (321h35m29.598020888s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1wR3ntox5QJSB1n45AmvQ94C6yzgihoBVYPMjsrgzTtxzhYHy1, address:45.91.116.4:4951): protocol: rpc: dial tcp 45.91.116.4:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136059113 (321h35m29.598151596s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: read tcp 188.166.225.170:41850->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136082528 (321h35m29.598175001s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: rpc: dial tcp 77.90.31.19:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136120630 (321h35m29.598213103s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX, address:45.91.116.6:4004): protocol: rpc: dial tcp 45.91.116.6:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136148143 (321h35m29.598240616s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: read tcp 188.166.225.170:47654->77.90.31.14:4669: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136173562 (321h35m29.598266035s) [seg_pos={0 0}, upload=283020]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 23:06:44.427092462 (321h35m29.889184945s) [upload=283020]: Done uploading segments... Oct 6 23:06:44.427128952 (321h35m29.889221435s) [upload=283020]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.595628185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..................X] duration: 1m0.293838795s FAIL (2025-10-06 23:06:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 21h34m30.318324337s) [upload=283020]: Got next segment. Inline: false Oct 6 23:05:44.856263134 (321h34m30.318355617s) [upload=283020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:44.856268845 (321h34m30.318361328s) [upload=283020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:45.108705808 (321h34m30.570798291s) [seg_pos={0 0}, upload=283020]: Begin upload segment... Oct 6 23:05:45.108853299 (321h34m30.570945772s) [seg_pos={0 0}, upload=283020]: Done begin upload segment. Oct 6 23:05:45.109033753 (321h34m30.571126226s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109109477 (321h34m30.571213192s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.109722314 (321h34m30.571814797s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110354308 (321h34m30.572446791s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.110950954 (321h34m30.573043437s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.111551888 (321h34m30.573644371s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112154475 (321h34m30.574246958s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.112932658 (321h34m30.575025141s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.113602423 (321h34m30.575694896s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.114401965 (321h34m30.576494448s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.116415860 (321h34m30.578508333s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.118906663 (321h34m30.580999136s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.121067647 (321h34m30.58316012s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.129477842 (321h34m30.591570325s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.130084146 (321h34m30.592176629s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.131174373 (321h34m30.593266866s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.133360686 (321h34m30.595453169s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:45.135350114 (321h34m30.597442597s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Uploading piece... Oct 6 23:05:46.574429033 (321h34m32.036521506s) [upload=283020]: Next returned nil segment Oct 6 23:05:46.574445103 (321h34m32.036537576s) [upload=283020]: Waiting for error group managing segments... Oct 6 23:05:46.946795634 (321h34m32.408888107s) [noise=true , address=145.79.227.43:4192 , piece_id=6JLTBQHEU3KKZ4OFWEQATMGTXE67OZFDVC7UM5RYA67ZUY7MPZCQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.970438174 (321h34m32.432530647s) [noise=true , address=145.79.227.42:4148 , piece_id=27XWQKN2JDRBA7DOKMHYQZOCC3HJANTXVMEFCKUZ4TH5QVRYY4QA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.971841325 (321h34m32.433933808s) [noise=true , address=145.79.227.41:4110 , piece_id=NZGGPKSFXGAE24Z27AVEEIQMUU7RO3PR2LZ6EURY5FG32ZHWWTQA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.982490243 (321h34m32.444582726s) [noise=true , address=145.79.227.25:4648 , piece_id=4E6R7ESUC674BTKOGW5YNIZRAY2OWS22IKZK4QZGLFHRPF6CXFHA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.990921268 (321h34m32.453013741s) [noise=true , address=145.79.227.32:4841 , piece_id=Z5OG5SFROZMPYENKND4XZYAWNGJPUUZ55MQWUOSBYSYLDK5EM4GA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.993986244 (321h34m32.456078727s) [noise=true , address=145.79.227.24:4586 , piece_id=RKURCMF2L2O4ZCETBAGP4ANGSSMGWOIC5PRFBYZNIMQ63FT2K6DQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:05:46.998097392 (321h34m32.460189875s) [noise=true , address=145.79.227.30:4795 , piece_id=F5EIYP6Z4X7PYUETRTFHITOWK26OMITC6KH4RUW6HCXLSLRETMSQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:24.533854514 (321h35m9.995946987s) [noise=true , address=77.90.31.18:4814 , piece_id=DHUEDW7F3JLMVHWXUG2XYMNMBKYBSXYE42ENKPYZS3W3ZMVD564A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.415951231 (321h35m10.878043704s) [noise=true , address=77.90.31.13:4625 , piece_id=BUYVH2YIMC2W4QFVKQCS5EJ6MG4WOSW5OY5SSFIPMT2EEUEI4UDQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:25.820324670 (321h35m11.282417143s) [noise=true , address=45.91.116.6:4984 , piece_id=NQD6H7WYAROVL55IJPTVIASHILHXMN354UMVXH55RJIEA755HA3A, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:27.866791732 (321h35m13.328884215s) [noise=true , address=45.91.116.14:4218 , piece_id=6MFJIYXILQLGQ3XPCHFXTFFNETWVH7JNALBJSCEW4WI6IQAMLBFA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:28.427216598 (321h35m13.889309071s) [noise=true , address=45.91.116.11:4144 , piece_id=ZYIOM37XZMSAPUD3N3ZEA4LMKP637XFBT7JZSEWAPU6A5VHPNETA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:35.312697524 (321h35m20.774789997s) [noise=true , address=77.90.31.21:4886 , piece_id=DK62EVPOPP57XMEFDY6YIAYB2CDFKNKINONR23AZFI7VKVL6UWZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: <nil> Oct 6 23:06:44.135928415 (321h35m29.598020888s) [noise=true , address=45.91.116.4:4951 , piece_id=FIZNZWWE4TRJUPU263MKGL6FEUFDSOJ6SF6RMJFVAYIFHRMICUEQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1wR3ntox5QJSB1n45AmvQ94C6yzgihoBVYPMjsrgzTtxzhYHy1, address:45.91.116.4:4951): protocol: rpc: dial tcp 45.91.116.4:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136059113 (321h35m29.598151596s) [noise=true , address=45.91.116.10:4087 , piece_id=ZWM2GPLBI6PD4WVEB4NS4K6H7NJLLRDXHCYK5VMPGWA765BIQ6WQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: read tcp 188.166.225.170:41850->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136082528 (321h35m29.598175001s) [noise=true , address=77.90.31.19:4844 , piece_id=4MD6X56PSCMXXGQPKNOBJZ3WXDZWQ7GZ5PC3GK3BG5A7IAQSDOZA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: rpc: dial tcp 77.90.31.19:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136120630 (321h35m29.598213103s) [noise=true , address=45.91.116.6:4004 , piece_id=6EVVPSN4NECR2BYLCVZLGNHP6MNMIGPJ7VFDNKZLFROLNHIQC5SQ, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX, address:45.91.116.6:4004): protocol: rpc: dial tcp 45.91.116.6:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136148143 (321h35m29.598240616s) [noise=true , address=77.90.31.14:4669 , piece_id=PIO5JZ7ZLPNMK4C6PJSCYXD4VMUK7XDPZIRXH4HVILFVWEOKQKMA, seg_pos={0 0}, upload=283020]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: read tcp 188.166.225.170:47654->77.90.31.14:4669: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:44.136173562 (321h35m29.598266035s) [seg_pos={0 0}, upload=283020]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 23:06:44.427092462 (321h35m29.889184945s) [upload=283020]: Done uploading segments... Oct 6 23:06:44.427128952 (321h35m29.889221435s) [upload=283020]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.293876267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X...........XX.......X] duration: 30.266092333s FAIL (2025-10-06 23:06:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:05:56.416568715 (321h34m41.878661188s) [upload=283023]: Starting upload Oct 6 23:05:56.416662404 (321h34m41.878754877s) [upload=283023]: Done starting upload Oct 6 23:05:56.417037158 (321h34m41.879129641s) [upload=283023]: Uploading segments... Oct 6 23:05:56.417096550 (321h34m41.879189023s) [upload=283023]: Got next segment. Inline: false Oct 6 23:05:56.417129693 (321h34m41.879222166s) [upload=283023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:05:56.417134293 (321h34m41.879226776s) [upload=283023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:05:56.670774287 (321h34m42.13286676s) [seg_pos={0 0}, upload=283023]: Begin upload segment... Oct 6 23:05:56.670846254 (321h34m42.132938727s) [seg_pos={0 0}, upload=283023]: Done begin upload segment. Oct 6 23:05:56.670877513 (321h34m42.132969987s) [noise=true , address=77.90.31.10:4517 , piece_id=TZAUEVDKXMXPU2G6QV5CNC2UTACQS4BY33EXIKHQEXAFEU3GI2CA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.671620058 (321h34m42.133712541s) [noise=true , address=45.91.116.8:4048 , piece_id=DNWNNHTKCKDN2JD4TJHI5DNUCYCDKQADSRMOYDLPLRCP6FYC4KLQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.671940328 (321h34m42.134032811s) [noise=true , address=145.79.227.3:4151 , piece_id=3PJVRPTKGC6HZRGNYX4XQKNF77XYVGUORMN7MZVDPPXOEA6OPRIQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.672214861 (321h34m42.134307334s) [noise=true , address=45.91.116.14:4190 , piece_id=4FWYOHPF2YGKVBUARUUKIC677B75P4MKCMBVDJAX42WBRCS4KAZQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.672588331 (321h34m42.134680814s) [noise=true , address=145.79.227.11:4732 , piece_id=N27D6TT34JUUBKZG4EEW6BKCLJ7SYQXQTHPYGPRYCCONCPAFJBZA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673102771 (321h34m42.135195254s) [noise=true , address=45.91.116.12:4225 , piece_id=VO2YOOXDFK5RRIZKTD5ML5G73KONYZBH766OGHF3L66TEM2QA2LQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673502983 (321h34m42.135595476s) [noise=true , address=77.90.31.13:4628 , piece_id=LVFG563ROTB7NVWKM4ICZ6WRVAM3SAMF3SDE2D72FDR52PPF2UHA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.673964462 (321h34m42.136056945s) [noise=true , address=145.79.227.2:4114 , piece_id=FTD3NMHADI245UVXUPNMGZEB2M3VJYKSBIT3MNWHG4IY5EPPKGQA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.675410636 (321h34m42.137503109s) [noise=true , address=45.91.116.9:4263 , piece_id=YYVJROMPN2FZU3S3EHQT74DU24JJ6YZU3FPT6I5JUPNOYVUXW5OA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.676852271 (321h34m42.138944754s) [noise=true , address=45.91.116.5:4298 , piece_id=4OBCY25BUZOQJJMGI6S4LXWRPUTG7FJTJYSYICSPIWUG2IJNBVTA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.680370049 (321h34m42.142462522s) [noise=true , address=145.79.227.14:4837 , piece_id=4VAZSD6ZX4HRF3HETNZR56ZFHSP6ZOOYXRBXQTEAYGFQNXJOFAUQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.682234680 (321h34m42.144327163s) [noise=true , address=145.79.227.15:4874 , piece_id=POV5RP5RPFLI4FILLPQ3TWW3O5C76U6ISO3ITILOOPSQFJDSPJIA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.683379109 (321h34m42.145471582s) [noise=true , address=145.79.227.5:4221 , piece_id=IK6CBA5POZ3TICDOPQSD73IWH3QZQRTECCTYG6KQC2IYY7AV7U4Q, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.683555826 (321h34m42.145648309s) [noise=true , address=145.79.227.17:4970 , piece_id=DWBWSXLIYNTH6PHNZ5JJWU5XJAEX3XXVHPZVTTTN6VUUZUZWJ54Q, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.685705879 (321h34m42.147798362s) [noise=true , address=145.79.227.4:4187 , piece_id=SPMGKOXWH2OOOSTR7UHVFSG64BO3IKFDGYFOQBJA2JJ65JLBH2UA, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:56.686479692 (321h34m42.148572167s) [noise=true , address=145.79.227.16:4911 , piece_id=JYL3UV2P3EEYQFZB5M2EKBWE4SUZOLPEFKGQ6NHDHGIQLESBYAIQ, seg_pos={0 0}, upload=283023]: Uploading piece... Oct 6 23:05:58.364067654 (321h34m43.826160137s) [upload=283023]: Next returned nil segment Oct 6 23:05:58.364086951 (321h34m43.826179434s) [upload=283023]: Waiting for error group managing segments... Oct 6 23:05:58.651101546 (321h34m44.11319404s) [noise=true , address=145.79.227.5:4221 , piece_id=IK6CBA5POZ3TICDOPQSD73IWH3QZQRTECCTYG6KQC2IYY7AV7U4Q, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.662711564 (321h34m44.124804047s) [noise=true , address=145.79.227.4:4187 , piece_id=SPMGKOXWH2OOOSTR7UHVFSG64BO3IKFDGYFOQBJA2JJ65JLBH2UA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.679699713 (321h34m44.141792196s) [noise=true , address=145.79.227.11:4732 , piece_id=N27D6TT34JUUBKZG4EEW6BKCLJ7SYQXQTHPYGPRYCCONCPAFJBZA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.679802989 (321h34m44.141895472s) [noise=true , address=145.79.227.2:4114 , piece_id=FTD3NMHADI245UVXUPNMGZEB2M3VJYKSBIT3MNWHG4IY5EPPKGQA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.691271527 (321h34m44.153364s) [noise=true , address=145.79.227.16:4911 , piece_id=JYL3UV2P3EEYQFZB5M2EKBWE4SUZOLPEFKGQ6NHDHGIQLESBYAIQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.692417900 (321h34m44.154510383s) [noise=true , address=145.79.227.17:4970 , piece_id=DWBWSXLIYNTH6PHNZ5JJWU5XJAEX3XXVHPZVTTTN6VUUZUZWJ54Q, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.694061870 (321h34m44.156154353s) [noise=true , address=145.79.227.15:4874 , piece_id=POV5RP5RPFLI4FILLPQ3TWW3O5C76U6ISO3ITILOOPSQFJDSPJIA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.707610320 (321h34m44.169702803s) [noise=true , address=145.79.227.3:4151 , piece_id=3PJVRPTKGC6HZRGNYX4XQKNF77XYVGUORMN7MZVDPPXOEA6OPRIQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:05:58.735344851 (321h34m44.197437324s) [noise=true , address=145.79.227.14:4837 , piece_id=4VAZSD6ZX4HRF3HETNZR56ZFHSP6ZOOYXRBXQTEAYGFQNXJOFAUQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: <nil> Oct 6 23:06:25.703533147 (321h35m11.16562563s) [noise=true , address=77.90.31.10:4517 , piece_id=TZAUEVDKXMXPU2G6QV5CNC2UTACQS4BY33EXIKHQEXAFEU3GI2CA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:36946->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703630603 (321h35m11.165723086s) [noise=true , address=45.91.116.8:4048 , piece_id=DNWNNHTKCKDN2JD4TJHI5DNUCYCDKQADSRMOYDLPLRCP6FYC4KLQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:48728->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703721517 (321h35m11.16581399s) [noise=true , address=45.91.116.14:4190 , piece_id=4FWYOHPF2YGKVBUARUUKIC677B75P4MKCMBVDJAX42WBRCS4KAZQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52264->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703801208 (321h35m11.165893681s) [noise=true , address=45.91.116.9:4263 , piece_id=YYVJROMPN2FZU3S3EHQT74DU24JJ6YZU3FPT6I5JUPNOYVUXW5OA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:34600->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:06:25.703948989 (321h35m11.166041462s) [noise=true , address=45.91.116.5:4298 , piece_id=4OBCY25BUZOQJJMGI6S4LXWRPUTG7FJTJYSYICSPIWUG2IJNBVTA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704020986 (321h35m11.166113459s) [noise=true , address=77.90.31.13:4628 , piece_id=LVFG563ROTB7NVWKM4ICZ6WRVAM3SAMF3SDE2D72FDR52PPF2UHA, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704125335 (321h35m11.166217818s) [noise=true , address=45.91.116.12:4225 , piece_id=VO2YOOXDFK5RRIZKTD5ML5G73KONYZBH766OGHF3L66TEM2QA2LQ, seg_pos={0 0}, upload=283023]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:06:25.704149722 (321h35m11.166242205s) [seg_pos={0 0}, upload=283023]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:06:25.968228364 (321h35m11.430320837s) [upload=283023]: Done uploading segments... Oct 6 23:06:25.968256157 (321h35m11.43034863s) [upload=283023]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.266115076s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XX........X........] duration: 6.512774215s FAIL (2025-10-06 23:06:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.512802069s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................XX........X.......] duration: 6.216832781s FAIL (2025-10-06 23:05:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.216865304s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X...........XX.......] duration: 6.612303992s FAIL (2025-10-06 23:05:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:59:54.702014331 (321h28m40.164106814s) [upload=282933]: Starting upload Oct 6 22:59:54.702104343 (321h28m40.164196826s) [upload=282933]: Done starting upload Oct 6 22:59:54.702222057 (321h28m40.16431454s) [upload=282933]: Uploading segments... Oct 6 22:59:54.702333680 (321h28m40.164426163s) [upload=282933]: Got next segment. Inline: false Oct 6 22:59:54.702403693 (321h28m40.164496176s) [upload=282933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:54.702411007 (321h28m40.16450349s) [upload=282933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:54.950723835 (321h28m40.412816318s) [seg_pos={0 0}, upload=282933]: Begin upload segment... Oct 6 22:59:54.950832923 (321h28m40.412925396s) [seg_pos={0 0}, upload=282933]: Done begin upload segment. Oct 6 22:59:54.950894781 (321h28m40.412987254s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951428066 (321h28m40.413520549s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951701657 (321h28m40.41379414s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952298664 (321h28m40.414391157s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952690530 (321h28m40.414783013s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952759842 (321h28m40.414852325s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953268962 (321h28m40.415361445s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953386826 (321h28m40.415479309s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953713879 (321h28m40.415806362s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954136494 (321h28m40.416228977s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954216355 (321h28m40.416308828s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954682753 (321h28m40.416775236s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954800929 (321h28m40.416893412s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955387456 (321h28m40.417479929s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955987779 (321h28m40.418080262s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.956564075 (321h28m40.418656558s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:56.570603436 (321h28m42.032695919s) [upload=282933]: Next returned nil segment Oct 6 22:59:56.570626519 (321h28m42.032719002s) [upload=282933]: Waiting for error group managing segments... Oct 6 22:59:56.928754072 (321h28m42.390846555s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.934964016 (321h28m42.39705651s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.945640235 (321h28m42.407732718s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.947979349 (321h28m42.410071823s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949424241 (321h28m42.411516724s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949861804 (321h28m42.411954287s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949972695 (321h28m42.412065178s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.951742295 (321h28m42.413834778s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.953872662 (321h28m42.415965145s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.954076059 (321h28m42.416168542s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.970066137 (321h28m42.43215861s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:23.983933742 (321h29m9.446026235s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:24.005052134 (321h29m9.467144637s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005233569 (321h29m9.467326052s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:48964->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005394526 (321h29m9.467487009s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59606->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005545624 (321h29m9.467638137s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:34112->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005581662 (321h29m9.467674145s) [seg_pos={0 0}, upload=282933]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 23:00:24.254694693 (321h29m9.716787176s) [upload=282933]: Done uploading segments... Oct 6 23:00:24.254730291 (321h29m9.716822764s) [upload=282933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.612339539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.096639269s FAIL (2025-10-06 23:04:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.096669957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X...........XX......] duration: 6.320160324s FAIL (2025-10-06 23:04:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:59:54.702014331 (321h28m40.164106814s) [upload=282933]: Starting upload Oct 6 22:59:54.702104343 (321h28m40.164196826s) [upload=282933]: Done starting upload Oct 6 22:59:54.702222057 (321h28m40.16431454s) [upload=282933]: Uploading segments... Oct 6 22:59:54.702333680 (321h28m40.164426163s) [upload=282933]: Got next segment. Inline: false Oct 6 22:59:54.702403693 (321h28m40.164496176s) [upload=282933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:54.702411007 (321h28m40.16450349s) [upload=282933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:54.950723835 (321h28m40.412816318s) [seg_pos={0 0}, upload=282933]: Begin upload segment... Oct 6 22:59:54.950832923 (321h28m40.412925396s) [seg_pos={0 0}, upload=282933]: Done begin upload segment. Oct 6 22:59:54.950894781 (321h28m40.412987254s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951428066 (321h28m40.413520549s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951701657 (321h28m40.41379414s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952298664 (321h28m40.414391157s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952690530 (321h28m40.414783013s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952759842 (321h28m40.414852325s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953268962 (321h28m40.415361445s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953386826 (321h28m40.415479309s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953713879 (321h28m40.415806362s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954136494 (321h28m40.416228977s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954216355 (321h28m40.416308828s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954682753 (321h28m40.416775236s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954800929 (321h28m40.416893412s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955387456 (321h28m40.417479929s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955987779 (321h28m40.418080262s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.956564075 (321h28m40.418656558s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:56.570603436 (321h28m42.032695919s) [upload=282933]: Next returned nil segment Oct 6 22:59:56.570626519 (321h28m42.032719002s) [upload=282933]: Waiting for error group managing segments... Oct 6 22:59:56.928754072 (321h28m42.390846555s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.934964016 (321h28m42.39705651s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.945640235 (321h28m42.407732718s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.947979349 (321h28m42.410071823s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949424241 (321h28m42.411516724s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949861804 (321h28m42.411954287s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949972695 (321h28m42.412065178s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.951742295 (321h28m42.413834778s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.953872662 (321h28m42.415965145s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.954076059 (321h28m42.416168542s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.970066137 (321h28m42.43215861s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:23.983933742 (321h29m9.446026235s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:24.005052134 (321h29m9.467144637s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005233569 (321h29m9.467326052s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:48964->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005394526 (321h29m9.467487009s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59606->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005545624 (321h29m9.467638137s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:34112->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005581662 (321h29m9.467674145s) [seg_pos={0 0}, upload=282933]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 23:00:24.254694693 (321h29m9.716787176s) [upload=282933]: Done uploading segments... Oct 6 23:00:24.254730291 (321h29m9.716822764s) [upload=282933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.320199398s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................XX........X......] duration: 6.397663591s FAIL (2025-10-06 23:04:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.397692235s edge gateway us-select-1

1
...
5691
5692
5693
5694
5695
...
9866