Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 6.191261276s FAIL (2025-09-22 04:40:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.191292816s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X.X.X....X......XXXXXXX.X......X.X.X..X.......X.XX.X...] duration: 6.286964642s FAIL (2025-09-22 04:40:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.286983969s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......XX..XX..X.X....XX........X.....X...X......X.X....X...] duration: 6.565621409s FAIL (2025-09-22 04:40:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 04:38:10.942153755 (375h9m46.981726885s) [upload=329754]: Starting upload Sep 22 04:38:10.942237284 (375h9m46.981810404s) [upload=329754]: Done starting upload Sep 22 04:38:10.942303550 (375h9m46.9818767s) [upload=329754]: Uploading segments... Sep 22 04:38:10.942541343 (375h9m46.982114473s) [upload=329754]: Got next segment. Inline: false Sep 22 04:38:10.942582341 (375h9m46.982155471s) [upload=329754]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:38:10.942589255 (375h9m46.982162385s) [upload=329754]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:38:11.198331095 (375h9m47.237904405s) [seg_pos={0 0}, upload=329754]: Begin upload segment... Sep 22 04:38:11.198405316 (375h9m47.237978446s) [seg_pos={0 0}, upload=329754]: Done begin upload segment. Sep 22 04:38:11.198440743 (375h9m47.238013873s) [noise=true , address=77.90.31.18:4806 , piece_id=I43GKMJ6MG7ZDLQIKOZOCNRQWOZLDCQWYEAIY6OQN33UIB4ZJSCQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198465371 (375h9m47.238038491s) [noise=true , address=145.79.227.7:4294 , piece_id=D7YTJ7W72IPO6G2KRORMSSS7UGFNGT77NJ35X73YXM2UJWI3OJQA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198949513 (375h9m47.238522633s) [noise=true , address=77.90.31.14:4661 , piece_id=2ZXSOYH6YVRYMGFT2JIPFOCZGY5UTVYCTFTNGGRUGUHRA6X3UZJQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198958079 (375h9m47.238531209s) [noise=true , address=45.91.116.14:4191 , piece_id=2RZ27526XGDJM76NOMH5DMQPU6CVFQBW7AXYLR7UIW3KMCZXWWQQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199409749 (375h9m47.238982879s) [noise=true , address=45.91.116.6:4974 , piece_id=CRIDN2F4VZNHJAYLFS6AUZNZMWMFWNXV52D5GJXDC2WP77GBOJAQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199423114 (375h9m47.238996234s) [noise=true , address=145.79.227.3:4149 , piece_id=DHFCMXM5B3AZZ5JUL5S73PNPBIHJS6YH7QL643VTE6K54AJPKE7A, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199810331 (375h9m47.239383461s) [noise=true , address=45.91.116.14:4189 , piece_id=BK2DCE6U4SOUDNVJF3RFURNCXIRE4YDCLFP7AROB4RXBEVQ2NMDQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199821984 (375h9m47.239395114s) [noise=true , address=145.79.227.18:4006 , piece_id=YZENL32HC2BW6R23NQDBDMMTWT5ECUFEOJNMV2U7HSV5ECVHK67Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200209311 (375h9m47.239782441s) [noise=true , address=77.90.31.17:4770 , piece_id=FJ2DIBUYCN5PBQLWZ7LPQBV3HRPUNRGZC55IHGKOGLGO2VWT3LXA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200292209 (375h9m47.239865339s) [noise=true , address=45.91.116.11:4118 , piece_id=6CDMYMM53ZTWDLJZZQS6NCLILKJAZBTR3HRAACVSIGKQBHG72WGA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200799154 (375h9m47.240372284s) [noise=true , address=45.91.116.5:4298 , piece_id=Z7LUKLSNTKVAWMDTDZROA5JWUE23NZCCMLAQBSBUHOR5ZS5JXSWA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200823050 (375h9m47.24039618s) [noise=true , address=77.90.31.11:4554 , piece_id=MYQXB53V46P6TR7A3AUH3BV45D7EQ7JER6YWD74C5ZDM7JYT2VZA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201232689 (375h9m47.24080581s) [noise=true , address=77.90.31.16:4733 , piece_id=P7LJX4IXADLUW6ZL5JXBUGKQQSLDOBK2VQKJOMYBOMR7U7BWEV5Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201345715 (375h9m47.240918846s) [noise=true , address=45.91.116.13:4153 , piece_id=OEG6Q4AJIVCYDRNOIT3CW7FSY5SPJ7GLMMD7XF5YNKE7XTI4IQ2Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201610299 (375h9m47.241183429s) [noise=true , address=145.79.227.16:4912 , piece_id=LJJPLGQJ7DLFFXBQPCP3FVCHAQX4XNYKNBX2WVJAHKHVASA7MRPQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.202776149 (375h9m47.242349279s) [noise=true , address=145.79.227.15:4875 , piece_id=EYO6PMH62OHVDRVUBJ7QQO4GOGLTRUN7SEJW3L6PO6E7XG5CMEBQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:12.453452126 (375h9m48.493025256s) [noise=true , address=145.79.227.3:4149 , piece_id=DHFCMXM5B3AZZ5JUL5S73PNPBIHJS6YH7QL643VTE6K54AJPKE7A, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:12.836211443 (375h9m48.875784573s) [upload=329754]: Next returned nil segment Sep 22 04:38:12.836242662 (375h9m48.875815792s) [upload=329754]: Waiting for error group managing segments... Sep 22 04:38:13.221522780 (375h9m49.26109591s) [noise=true , address=77.90.31.17:4770 , piece_id=FJ2DIBUYCN5PBQLWZ7LPQBV3HRPUNRGZC55IHGKOGLGO2VWT3LXA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.248245675 (375h9m49.287818796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRIDN2F4VZNHJAYLFS6AUZNZMWMFWNXV52D5GJXDC2WP77GBOJAQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.248820760 (375h9m49.28839389s) [noise=true , address=77.90.31.14:4661 , piece_id=2ZXSOYH6YVRYMGFT2JIPFOCZGY5UTVYCTFTNGGRUGUHRA6X3UZJQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.257583155 (375h9m49.297156285s) [noise=true , address=45.91.116.14:4189 , piece_id=BK2DCE6U4SOUDNVJF3RFURNCXIRE4YDCLFP7AROB4RXBEVQ2NMDQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.300265320 (375h9m49.339838451s) [noise=true , address=77.90.31.16:4733 , piece_id=P7LJX4IXADLUW6ZL5JXBUGKQQSLDOBK2VQKJOMYBOMR7U7BWEV5Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.303358921 (375h9m49.342932041s) [noise=true , address=45.91.116.14:4191 , piece_id=2RZ27526XGDJM76NOMH5DMQPU6CVFQBW7AXYLR7UIW3KMCZXWWQQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.310621631 (375h9m49.350194761s) [noise=true , address=45.91.116.13:4153 , piece_id=OEG6Q4AJIVCYDRNOIT3CW7FSY5SPJ7GLMMD7XF5YNKE7XTI4IQ2Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.319846096 (375h9m49.359419226s) [noise=true , address=77.90.31.11:4554 , piece_id=MYQXB53V46P6TR7A3AUH3BV45D7EQ7JER6YWD74C5ZDM7JYT2VZA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.335515573 (375h9m49.375088703s) [noise=true , address=45.91.116.11:4118 , piece_id=6CDMYMM53ZTWDLJZZQS6NCLILKJAZBTR3HRAACVSIGKQBHG72WGA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.337850670 (375h9m49.3774238s) [noise=true , address=45.91.116.5:4298 , piece_id=Z7LUKLSNTKVAWMDTDZROA5JWUE23NZCCMLAQBSBUHOR5ZS5JXSWA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.340727518 (375h9m49.380300648s) [noise=true , address=77.90.31.18:4806 , piece_id=I43GKMJ6MG7ZDLQIKOZOCNRQWOZLDCQWYEAIY6OQN33UIB4ZJSCQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:40.221051988 (375h10m16.260625108s) [noise=true , address=145.79.227.18:4006 , piece_id=YZENL32HC2BW6R23NQDBDMMTWT5ECUFEOJNMV2U7HSV5ECVHK67Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:40.221123283 (375h10m16.260696403s) [noise=true , address=145.79.227.7:4294 , piece_id=D7YTJ7W72IPO6G2KRORMSSS7UGFNGT77NJ35X73YXM2UJWI3OJQA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:40.221240136 (375h10m16.260813267s) [noise=true , address=145.79.227.15:4875 , piece_id=EYO6PMH62OHVDRVUBJ7QQO4GOGLTRUN7SEJW3L6PO6E7XG5CMEBQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:38:40.221385924 (375h10m16.260959054s) [noise=true , address=145.79.227.16:4912 , piece_id=LJJPLGQJ7DLFFXBQPCP3FVCHAQX4XNYKNBX2WVJAHKHVASA7MRPQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:38:40.221402926 (375h10m16.260976056s) [seg_pos={0 0}, upload=329754]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 04:38:40.469513216 (375h10m16.509086346s) [upload=329754]: Done uploading segments... Sep 22 04:38:40.469548624 (375h10m16.509121754s) [upload=329754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.565659211s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X.X.X....X......XXXXXXX.X......X.X.X..X.......X.XX.X..] duration: 6.325666879s FAIL (2025-09-22 04:40:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.325712656s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 6.162007322s FAIL (2025-09-22 04:40:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.162046417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........XX..XX..X.X....XX........X.....X...X......X.X....X..] duration: 6.188160676s FAIL (2025-09-22 04:39:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 04:38:10.942153755 (375h9m46.981726885s) [upload=329754]: Starting upload Sep 22 04:38:10.942237284 (375h9m46.981810404s) [upload=329754]: Done starting upload Sep 22 04:38:10.942303550 (375h9m46.9818767s) [upload=329754]: Uploading segments... Sep 22 04:38:10.942541343 (375h9m46.982114473s) [upload=329754]: Got next segment. Inline: false Sep 22 04:38:10.942582341 (375h9m46.982155471s) [upload=329754]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:38:10.942589255 (375h9m46.982162385s) [upload=329754]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:38:11.198331095 (375h9m47.237904405s) [seg_pos={0 0}, upload=329754]: Begin upload segment... Sep 22 04:38:11.198405316 (375h9m47.237978446s) [seg_pos={0 0}, upload=329754]: Done begin upload segment. Sep 22 04:38:11.198440743 (375h9m47.238013873s) [noise=true , address=77.90.31.18:4806 , piece_id=I43GKMJ6MG7ZDLQIKOZOCNRQWOZLDCQWYEAIY6OQN33UIB4ZJSCQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198465371 (375h9m47.238038491s) [noise=true , address=145.79.227.7:4294 , piece_id=D7YTJ7W72IPO6G2KRORMSSS7UGFNGT77NJ35X73YXM2UJWI3OJQA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198949513 (375h9m47.238522633s) [noise=true , address=77.90.31.14:4661 , piece_id=2ZXSOYH6YVRYMGFT2JIPFOCZGY5UTVYCTFTNGGRUGUHRA6X3UZJQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198958079 (375h9m47.238531209s) [noise=true , address=45.91.116.14:4191 , piece_id=2RZ27526XGDJM76NOMH5DMQPU6CVFQBW7AXYLR7UIW3KMCZXWWQQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199409749 (375h9m47.238982879s) [noise=true , address=45.91.116.6:4974 , piece_id=CRIDN2F4VZNHJAYLFS6AUZNZMWMFWNXV52D5GJXDC2WP77GBOJAQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199423114 (375h9m47.238996234s) [noise=true , address=145.79.227.3:4149 , piece_id=DHFCMXM5B3AZZ5JUL5S73PNPBIHJS6YH7QL643VTE6K54AJPKE7A, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199810331 (375h9m47.239383461s) [noise=true , address=45.91.116.14:4189 , piece_id=BK2DCE6U4SOUDNVJF3RFURNCXIRE4YDCLFP7AROB4RXBEVQ2NMDQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199821984 (375h9m47.239395114s) [noise=true , address=145.79.227.18:4006 , piece_id=YZENL32HC2BW6R23NQDBDMMTWT5ECUFEOJNMV2U7HSV5ECVHK67Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200209311 (375h9m47.239782441s) [noise=true , address=77.90.31.17:4770 , piece_id=FJ2DIBUYCN5PBQLWZ7LPQBV3HRPUNRGZC55IHGKOGLGO2VWT3LXA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200292209 (375h9m47.239865339s) [noise=true , address=45.91.116.11:4118 , piece_id=6CDMYMM53ZTWDLJZZQS6NCLILKJAZBTR3HRAACVSIGKQBHG72WGA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200799154 (375h9m47.240372284s) [noise=true , address=45.91.116.5:4298 , piece_id=Z7LUKLSNTKVAWMDTDZROA5JWUE23NZCCMLAQBSBUHOR5ZS5JXSWA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200823050 (375h9m47.24039618s) [noise=true , address=77.90.31.11:4554 , piece_id=MYQXB53V46P6TR7A3AUH3BV45D7EQ7JER6YWD74C5ZDM7JYT2VZA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201232689 (375h9m47.24080581s) [noise=true , address=77.90.31.16:4733 , piece_id=P7LJX4IXADLUW6ZL5JXBUGKQQSLDOBK2VQKJOMYBOMR7U7BWEV5Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201345715 (375h9m47.240918846s) [noise=true , address=45.91.116.13:4153 , piece_id=OEG6Q4AJIVCYDRNOIT3CW7FSY5SPJ7GLMMD7XF5YNKE7XTI4IQ2Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201610299 (375h9m47.241183429s) [noise=true , address=145.79.227.16:4912 , piece_id=LJJPLGQJ7DLFFXBQPCP3FVCHAQX4XNYKNBX2WVJAHKHVASA7MRPQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.202776149 (375h9m47.242349279s) [noise=true , address=145.79.227.15:4875 , piece_id=EYO6PMH62OHVDRVUBJ7QQO4GOGLTRUN7SEJW3L6PO6E7XG5CMEBQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:12.453452126 (375h9m48.493025256s) [noise=true , address=145.79.227.3:4149 , piece_id=DHFCMXM5B3AZZ5JUL5S73PNPBIHJS6YH7QL643VTE6K54AJPKE7A, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:12.836211443 (375h9m48.875784573s) [upload=329754]: Next returned nil segment Sep 22 04:38:12.836242662 (375h9m48.875815792s) [upload=329754]: Waiting for error group managing segments... Sep 22 04:38:13.221522780 (375h9m49.26109591s) [noise=true , address=77.90.31.17:4770 , piece_id=FJ2DIBUYCN5PBQLWZ7LPQBV3HRPUNRGZC55IHGKOGLGO2VWT3LXA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.248245675 (375h9m49.287818796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRIDN2F4VZNHJAYLFS6AUZNZMWMFWNXV52D5GJXDC2WP77GBOJAQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.248820760 (375h9m49.28839389s) [noise=true , address=77.90.31.14:4661 , piece_id=2ZXSOYH6YVRYMGFT2JIPFOCZGY5UTVYCTFTNGGRUGUHRA6X3UZJQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.257583155 (375h9m49.297156285s) [noise=true , address=45.91.116.14:4189 , piece_id=BK2DCE6U4SOUDNVJF3RFURNCXIRE4YDCLFP7AROB4RXBEVQ2NMDQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.300265320 (375h9m49.339838451s) [noise=true , address=77.90.31.16:4733 , piece_id=P7LJX4IXADLUW6ZL5JXBUGKQQSLDOBK2VQKJOMYBOMR7U7BWEV5Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.303358921 (375h9m49.342932041s) [noise=true , address=45.91.116.14:4191 , piece_id=2RZ27526XGDJM76NOMH5DMQPU6CVFQBW7AXYLR7UIW3KMCZXWWQQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.310621631 (375h9m49.350194761s) [noise=true , address=45.91.116.13:4153 , piece_id=OEG6Q4AJIVCYDRNOIT3CW7FSY5SPJ7GLMMD7XF5YNKE7XTI4IQ2Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.319846096 (375h9m49.359419226s) [noise=true , address=77.90.31.11:4554 , piece_id=MYQXB53V46P6TR7A3AUH3BV45D7EQ7JER6YWD74C5ZDM7JYT2VZA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.335515573 (375h9m49.375088703s) [noise=true , address=45.91.116.11:4118 , piece_id=6CDMYMM53ZTWDLJZZQS6NCLILKJAZBTR3HRAACVSIGKQBHG72WGA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.337850670 (375h9m49.3774238s) [noise=true , address=45.91.116.5:4298 , piece_id=Z7LUKLSNTKVAWMDTDZROA5JWUE23NZCCMLAQBSBUHOR5ZS5JXSWA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.340727518 (375h9m49.380300648s) [noise=true , address=77.90.31.18:4806 , piece_id=I43GKMJ6MG7ZDLQIKOZOCNRQWOZLDCQWYEAIY6OQN33UIB4ZJSCQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:40.221051988 (375h10m16.260625108s) [noise=true , address=145.79.227.18:4006 , piece_id=YZENL32HC2BW6R23NQDBDMMTWT5ECUFEOJNMV2U7HSV5ECVHK67Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:40.221123283 (375h10m16.260696403s) [noise=true , address=145.79.227.7:4294 , piece_id=D7YTJ7W72IPO6G2KRORMSSS7UGFNGT77NJ35X73YXM2UJWI3OJQA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:40.221240136 (375h10m16.260813267s) [noise=true , address=145.79.227.15:4875 , piece_id=EYO6PMH62OHVDRVUBJ7QQO4GOGLTRUN7SEJW3L6PO6E7XG5CMEBQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:38:40.221385924 (375h10m16.260959054s) [noise=true , address=145.79.227.16:4912 , piece_id=LJJPLGQJ7DLFFXBQPCP3FVCHAQX4XNYKNBX2WVJAHKHVASA7MRPQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:38:40.221402926 (375h10m16.260976056s) [seg_pos={0 0}, upload=329754]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 04:38:40.469513216 (375h10m16.509086346s) [upload=329754]: Done uploading segments... Sep 22 04:38:40.469548624 (375h10m16.509121754s) [upload=329754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.188189211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........XX..XX..X.X....XX........X.....X...X......X.X....X.] duration: 6.272105749s FAIL (2025-09-22 04:39:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 04:38:10.942153755 (375h9m46.981726885s) [upload=329754]: Starting upload Sep 22 04:38:10.942237284 (375h9m46.981810404s) [upload=329754]: Done starting upload Sep 22 04:38:10.942303550 (375h9m46.9818767s) [upload=329754]: Uploading segments... Sep 22 04:38:10.942541343 (375h9m46.982114473s) [upload=329754]: Got next segment. Inline: false Sep 22 04:38:10.942582341 (375h9m46.982155471s) [upload=329754]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:38:10.942589255 (375h9m46.982162385s) [upload=329754]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:38:11.198331095 (375h9m47.237904405s) [seg_pos={0 0}, upload=329754]: Begin upload segment... Sep 22 04:38:11.198405316 (375h9m47.237978446s) [seg_pos={0 0}, upload=329754]: Done begin upload segment. Sep 22 04:38:11.198440743 (375h9m47.238013873s) [noise=true , address=77.90.31.18:4806 , piece_id=I43GKMJ6MG7ZDLQIKOZOCNRQWOZLDCQWYEAIY6OQN33UIB4ZJSCQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198465371 (375h9m47.238038491s) [noise=true , address=145.79.227.7:4294 , piece_id=D7YTJ7W72IPO6G2KRORMSSS7UGFNGT77NJ35X73YXM2UJWI3OJQA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198949513 (375h9m47.238522633s) [noise=true , address=77.90.31.14:4661 , piece_id=2ZXSOYH6YVRYMGFT2JIPFOCZGY5UTVYCTFTNGGRUGUHRA6X3UZJQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198958079 (375h9m47.238531209s) [noise=true , address=45.91.116.14:4191 , piece_id=2RZ27526XGDJM76NOMH5DMQPU6CVFQBW7AXYLR7UIW3KMCZXWWQQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199409749 (375h9m47.238982879s) [noise=true , address=45.91.116.6:4974 , piece_id=CRIDN2F4VZNHJAYLFS6AUZNZMWMFWNXV52D5GJXDC2WP77GBOJAQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199423114 (375h9m47.238996234s) [noise=true , address=145.79.227.3:4149 , piece_id=DHFCMXM5B3AZZ5JUL5S73PNPBIHJS6YH7QL643VTE6K54AJPKE7A, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199810331 (375h9m47.239383461s) [noise=true , address=45.91.116.14:4189 , piece_id=BK2DCE6U4SOUDNVJF3RFURNCXIRE4YDCLFP7AROB4RXBEVQ2NMDQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199821984 (375h9m47.239395114s) [noise=true , address=145.79.227.18:4006 , piece_id=YZENL32HC2BW6R23NQDBDMMTWT5ECUFEOJNMV2U7HSV5ECVHK67Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200209311 (375h9m47.239782441s) [noise=true , address=77.90.31.17:4770 , piece_id=FJ2DIBUYCN5PBQLWZ7LPQBV3HRPUNRGZC55IHGKOGLGO2VWT3LXA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200292209 (375h9m47.239865339s) [noise=true , address=45.91.116.11:4118 , piece_id=6CDMYMM53ZTWDLJZZQS6NCLILKJAZBTR3HRAACVSIGKQBHG72WGA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200799154 (375h9m47.240372284s) [noise=true , address=45.91.116.5:4298 , piece_id=Z7LUKLSNTKVAWMDTDZROA5JWUE23NZCCMLAQBSBUHOR5ZS5JXSWA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200823050 (375h9m47.24039618s) [noise=true , address=77.90.31.11:4554 , piece_id=MYQXB53V46P6TR7A3AUH3BV45D7EQ7JER6YWD74C5ZDM7JYT2VZA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201232689 (375h9m47.24080581s) [noise=true , address=77.90.31.16:4733 , piece_id=P7LJX4IXADLUW6ZL5JXBUGKQQSLDOBK2VQKJOMYBOMR7U7BWEV5Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201345715 (375h9m47.240918846s) [noise=true , address=45.91.116.13:4153 , piece_id=OEG6Q4AJIVCYDRNOIT3CW7FSY5SPJ7GLMMD7XF5YNKE7XTI4IQ2Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201610299 (375h9m47.241183429s) [noise=true , address=145.79.227.16:4912 , piece_id=LJJPLGQJ7DLFFXBQPCP3FVCHAQX4XNYKNBX2WVJAHKHVASA7MRPQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.202776149 (375h9m47.242349279s) [noise=true , address=145.79.227.15:4875 , piece_id=EYO6PMH62OHVDRVUBJ7QQO4GOGLTRUN7SEJW3L6PO6E7XG5CMEBQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:12.453452126 (375h9m48.493025256s) [noise=true , address=145.79.227.3:4149 , piece_id=DHFCMXM5B3AZZ5JUL5S73PNPBIHJS6YH7QL643VTE6K54AJPKE7A, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:12.836211443 (375h9m48.875784573s) [upload=329754]: Next returned nil segment Sep 22 04:38:12.836242662 (375h9m48.875815792s) [upload=329754]: Waiting for error group managing segments... Sep 22 04:38:13.221522780 (375h9m49.26109591s) [noise=true , address=77.90.31.17:4770 , piece_id=FJ2DIBUYCN5PBQLWZ7LPQBV3HRPUNRGZC55IHGKOGLGO2VWT3LXA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.248245675 (375h9m49.287818796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRIDN2F4VZNHJAYLFS6AUZNZMWMFWNXV52D5GJXDC2WP77GBOJAQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.248820760 (375h9m49.28839389s) [noise=true , address=77.90.31.14:4661 , piece_id=2ZXSOYH6YVRYMGFT2JIPFOCZGY5UTVYCTFTNGGRUGUHRA6X3UZJQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.257583155 (375h9m49.297156285s) [noise=true , address=45.91.116.14:4189 , piece_id=BK2DCE6U4SOUDNVJF3RFURNCXIRE4YDCLFP7AROB4RXBEVQ2NMDQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.300265320 (375h9m49.339838451s) [noise=true , address=77.90.31.16:4733 , piece_id=P7LJX4IXADLUW6ZL5JXBUGKQQSLDOBK2VQKJOMYBOMR7U7BWEV5Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.303358921 (375h9m49.342932041s) [noise=true , address=45.91.116.14:4191 , piece_id=2RZ27526XGDJM76NOMH5DMQPU6CVFQBW7AXYLR7UIW3KMCZXWWQQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.310621631 (375h9m49.350194761s) [noise=true , address=45.91.116.13:4153 , piece_id=OEG6Q4AJIVCYDRNOIT3CW7FSY5SPJ7GLMMD7XF5YNKE7XTI4IQ2Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.319846096 (375h9m49.359419226s) [noise=true , address=77.90.31.11:4554 , piece_id=MYQXB53V46P6TR7A3AUH3BV45D7EQ7JER6YWD74C5ZDM7JYT2VZA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.335515573 (375h9m49.375088703s) [noise=true , address=45.91.116.11:4118 , piece_id=6CDMYMM53ZTWDLJZZQS6NCLILKJAZBTR3HRAACVSIGKQBHG72WGA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.337850670 (375h9m49.3774238s) [noise=true , address=45.91.116.5:4298 , piece_id=Z7LUKLSNTKVAWMDTDZROA5JWUE23NZCCMLAQBSBUHOR5ZS5JXSWA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.340727518 (375h9m49.380300648s) [noise=true , address=77.90.31.18:4806 , piece_id=I43GKMJ6MG7ZDLQIKOZOCNRQWOZLDCQWYEAIY6OQN33UIB4ZJSCQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:40.221051988 (375h10m16.260625108s) [noise=true , address=145.79.227.18:4006 , piece_id=YZENL32HC2BW6R23NQDBDMMTWT5ECUFEOJNMV2U7HSV5ECVHK67Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:40.221123283 (375h10m16.260696403s) [noise=true , address=145.79.227.7:4294 , piece_id=D7YTJ7W72IPO6G2KRORMSSS7UGFNGT77NJ35X73YXM2UJWI3OJQA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:40.221240136 (375h10m16.260813267s) [noise=true , address=145.79.227.15:4875 , piece_id=EYO6PMH62OHVDRVUBJ7QQO4GOGLTRUN7SEJW3L6PO6E7XG5CMEBQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:38:40.221385924 (375h10m16.260959054s) [noise=true , address=145.79.227.16:4912 , piece_id=LJJPLGQJ7DLFFXBQPCP3FVCHAQX4XNYKNBX2WVJAHKHVASA7MRPQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:38:40.221402926 (375h10m16.260976056s) [seg_pos={0 0}, upload=329754]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 04:38:40.469513216 (375h10m16.509086346s) [upload=329754]: Done uploading segments... Sep 22 04:38:40.469548624 (375h10m16.509121754s) [upload=329754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.272141666s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.......X.X.X....X......XXXXXXX.X......X.X.X..X.......X.XX.X.] duration: 6.133414876s FAIL (2025-09-22 04:39:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.133435407s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 5.909357317s FAIL (2025-09-22 04:38:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.909390221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.......X.X.X....X......XXXXXXX.X......X.X.X..X.......X.XX.X] duration: 30.001374475s FAIL (2025-09-22 04:38:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001687993s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 6.193190513s FAIL (2025-09-22 04:38:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.193213658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.........XX..XX..X.X....XX........X.....X...X......X.X....X] duration: 30.250740046s FAIL (2025-09-22 04:38:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 04:38:10.942153755 (375h9m46.981726885s) [upload=329754]: Starting upload Sep 22 04:38:10.942237284 (375h9m46.981810404s) [upload=329754]: Done starting upload Sep 22 04:38:10.942303550 (375h9m46.9818767s) [upload=329754]: Uploading segments... Sep 22 04:38:10.942541343 (375h9m46.982114473s) [upload=329754]: Got next segment. Inline: false Sep 22 04:38:10.942582341 (375h9m46.982155471s) [upload=329754]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:38:10.942589255 (375h9m46.982162385s) [upload=329754]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:38:11.198331095 (375h9m47.237904405s) [seg_pos={0 0}, upload=329754]: Begin upload segment... Sep 22 04:38:11.198405316 (375h9m47.237978446s) [seg_pos={0 0}, upload=329754]: Done begin upload segment. Sep 22 04:38:11.198440743 (375h9m47.238013873s) [noise=true , address=77.90.31.18:4806 , piece_id=I43GKMJ6MG7ZDLQIKOZOCNRQWOZLDCQWYEAIY6OQN33UIB4ZJSCQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198465371 (375h9m47.238038491s) [noise=true , address=145.79.227.7:4294 , piece_id=D7YTJ7W72IPO6G2KRORMSSS7UGFNGT77NJ35X73YXM2UJWI3OJQA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198949513 (375h9m47.238522633s) [noise=true , address=77.90.31.14:4661 , piece_id=2ZXSOYH6YVRYMGFT2JIPFOCZGY5UTVYCTFTNGGRUGUHRA6X3UZJQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.198958079 (375h9m47.238531209s) [noise=true , address=45.91.116.14:4191 , piece_id=2RZ27526XGDJM76NOMH5DMQPU6CVFQBW7AXYLR7UIW3KMCZXWWQQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199409749 (375h9m47.238982879s) [noise=true , address=45.91.116.6:4974 , piece_id=CRIDN2F4VZNHJAYLFS6AUZNZMWMFWNXV52D5GJXDC2WP77GBOJAQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199423114 (375h9m47.238996234s) [noise=true , address=145.79.227.3:4149 , piece_id=DHFCMXM5B3AZZ5JUL5S73PNPBIHJS6YH7QL643VTE6K54AJPKE7A, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199810331 (375h9m47.239383461s) [noise=true , address=45.91.116.14:4189 , piece_id=BK2DCE6U4SOUDNVJF3RFURNCXIRE4YDCLFP7AROB4RXBEVQ2NMDQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.199821984 (375h9m47.239395114s) [noise=true , address=145.79.227.18:4006 , piece_id=YZENL32HC2BW6R23NQDBDMMTWT5ECUFEOJNMV2U7HSV5ECVHK67Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200209311 (375h9m47.239782441s) [noise=true , address=77.90.31.17:4770 , piece_id=FJ2DIBUYCN5PBQLWZ7LPQBV3HRPUNRGZC55IHGKOGLGO2VWT3LXA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200292209 (375h9m47.239865339s) [noise=true , address=45.91.116.11:4118 , piece_id=6CDMYMM53ZTWDLJZZQS6NCLILKJAZBTR3HRAACVSIGKQBHG72WGA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200799154 (375h9m47.240372284s) [noise=true , address=45.91.116.5:4298 , piece_id=Z7LUKLSNTKVAWMDTDZROA5JWUE23NZCCMLAQBSBUHOR5ZS5JXSWA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.200823050 (375h9m47.24039618s) [noise=true , address=77.90.31.11:4554 , piece_id=MYQXB53V46P6TR7A3AUH3BV45D7EQ7JER6YWD74C5ZDM7JYT2VZA, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201232689 (375h9m47.24080581s) [noise=true , address=77.90.31.16:4733 , piece_id=P7LJX4IXADLUW6ZL5JXBUGKQQSLDOBK2VQKJOMYBOMR7U7BWEV5Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201345715 (375h9m47.240918846s) [noise=true , address=45.91.116.13:4153 , piece_id=OEG6Q4AJIVCYDRNOIT3CW7FSY5SPJ7GLMMD7XF5YNKE7XTI4IQ2Q, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.201610299 (375h9m47.241183429s) [noise=true , address=145.79.227.16:4912 , piece_id=LJJPLGQJ7DLFFXBQPCP3FVCHAQX4XNYKNBX2WVJAHKHVASA7MRPQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:11.202776149 (375h9m47.242349279s) [noise=true , address=145.79.227.15:4875 , piece_id=EYO6PMH62OHVDRVUBJ7QQO4GOGLTRUN7SEJW3L6PO6E7XG5CMEBQ, seg_pos={0 0}, upload=329754]: Uploading piece... Sep 22 04:38:12.453452126 (375h9m48.493025256s) [noise=true , address=145.79.227.3:4149 , piece_id=DHFCMXM5B3AZZ5JUL5S73PNPBIHJS6YH7QL643VTE6K54AJPKE7A, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:12.836211443 (375h9m48.875784573s) [upload=329754]: Next returned nil segment Sep 22 04:38:12.836242662 (375h9m48.875815792s) [upload=329754]: Waiting for error group managing segments... Sep 22 04:38:13.221522780 (375h9m49.26109591s) [noise=true , address=77.90.31.17:4770 , piece_id=FJ2DIBUYCN5PBQLWZ7LPQBV3HRPUNRGZC55IHGKOGLGO2VWT3LXA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.248245675 (375h9m49.287818796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRIDN2F4VZNHJAYLFS6AUZNZMWMFWNXV52D5GJXDC2WP77GBOJAQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.248820760 (375h9m49.28839389s) [noise=true , address=77.90.31.14:4661 , piece_id=2ZXSOYH6YVRYMGFT2JIPFOCZGY5UTVYCTFTNGGRUGUHRA6X3UZJQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.257583155 (375h9m49.297156285s) [noise=true , address=45.91.116.14:4189 , piece_id=BK2DCE6U4SOUDNVJF3RFURNCXIRE4YDCLFP7AROB4RXBEVQ2NMDQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.300265320 (375h9m49.339838451s) [noise=true , address=77.90.31.16:4733 , piece_id=P7LJX4IXADLUW6ZL5JXBUGKQQSLDOBK2VQKJOMYBOMR7U7BWEV5Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.303358921 (375h9m49.342932041s) [noise=true , address=45.91.116.14:4191 , piece_id=2RZ27526XGDJM76NOMH5DMQPU6CVFQBW7AXYLR7UIW3KMCZXWWQQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.310621631 (375h9m49.350194761s) [noise=true , address=45.91.116.13:4153 , piece_id=OEG6Q4AJIVCYDRNOIT3CW7FSY5SPJ7GLMMD7XF5YNKE7XTI4IQ2Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.319846096 (375h9m49.359419226s) [noise=true , address=77.90.31.11:4554 , piece_id=MYQXB53V46P6TR7A3AUH3BV45D7EQ7JER6YWD74C5ZDM7JYT2VZA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.335515573 (375h9m49.375088703s) [noise=true , address=45.91.116.11:4118 , piece_id=6CDMYMM53ZTWDLJZZQS6NCLILKJAZBTR3HRAACVSIGKQBHG72WGA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.337850670 (375h9m49.3774238s) [noise=true , address=45.91.116.5:4298 , piece_id=Z7LUKLSNTKVAWMDTDZROA5JWUE23NZCCMLAQBSBUHOR5ZS5JXSWA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:13.340727518 (375h9m49.380300648s) [noise=true , address=77.90.31.18:4806 , piece_id=I43GKMJ6MG7ZDLQIKOZOCNRQWOZLDCQWYEAIY6OQN33UIB4ZJSCQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: <nil> Sep 22 04:38:40.221051988 (375h10m16.260625108s) [noise=true , address=145.79.227.18:4006 , piece_id=YZENL32HC2BW6R23NQDBDMMTWT5ECUFEOJNMV2U7HSV5ECVHK67Q, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:40.221123283 (375h10m16.260696403s) [noise=true , address=145.79.227.7:4294 , piece_id=D7YTJ7W72IPO6G2KRORMSSS7UGFNGT77NJ35X73YXM2UJWI3OJQA, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:40.221240136 (375h10m16.260813267s) [noise=true , address=145.79.227.15:4875 , piece_id=EYO6PMH62OHVDRVUBJ7QQO4GOGLTRUN7SEJW3L6PO6E7XG5CMEBQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:38:40.221385924 (375h10m16.260959054s) [noise=true , address=145.79.227.16:4912 , piece_id=LJJPLGQJ7DLFFXBQPCP3FVCHAQX4XNYKNBX2WVJAHKHVASA7MRPQ, seg_pos={0 0}, upload=329754]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:38:40.221402926 (375h10m16.260976056s) [seg_pos={0 0}, upload=329754]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 04:38:40.469513216 (375h10m16.509086346s) [upload=329754]: Done uploading segments... Sep 22 04:38:40.469548624 (375h10m16.509121754s) [upload=329754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.250774422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.........XX..XX..X.X....XX........X.....X...X......X.X....] duration: 6.143300391s FAIL (2025-09-22 04:37:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:35:34.758938037 (375h7m10.798511177s) [upload=329713]: Starting upload Sep 22 04:35:34.759019573 (375h7m10.798592703s) [upload=329713]: Done starting upload Sep 22 04:35:34.759083996 (375h7m10.798657116s) [upload=329713]: Uploading segments... Sep 22 04:35:34.759305898 (375h7m10.798879028s) [upload=329713]: Got next segment. Inline: false Sep 22 04:35:34.759347898 (375h7m10.798921028s) [upload=329713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:35:34.759354610 (375h7m10.79892777s) [upload=329713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:35:35.037917458 (375h7m11.077490588s) [seg_pos={0 0}, upload=329713]: Begin upload segment... Sep 22 04:35:35.037974858 (375h7m11.077547988s) [seg_pos={0 0}, upload=329713]: Done begin upload segment. Sep 22 04:35:35.037993022 (375h7m11.077566142s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038009483 (375h7m11.077582614s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038066441 (375h7m11.077639571s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038432619 (375h7m11.078005759s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038583556 (375h7m11.078156686s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038940667 (375h7m11.078513787s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039287507 (375h7m11.078860627s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039682158 (375h7m11.079255279s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040009171 (375h7m11.079582301s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040491379 (375h7m11.080064509s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040750273 (375h7m11.080323393s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041190100 (375h7m11.08076322s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041528865 (375h7m11.081101995s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041913267 (375h7m11.081486397s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043178876 (375h7m11.082752016s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043813055 (375h7m11.083386185s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:36.831646663 (375h7m12.871219793s) [upload=329713]: Next returned nil segment Sep 22 04:35:36.831673043 (375h7m12.871246173s) [upload=329713]: Waiting for error group managing segments... Sep 22 04:35:36.958845684 (375h7m12.998418804s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:35:37.139170497 (375h7m13.178743617s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.140993289 (375h7m13.180566419s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.141524941 (375h7m13.181098071s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.142292071 (375h7m13.181865202s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.168995379 (375h7m13.208568509s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.183756498 (375h7m13.223329618s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.196934151 (375h7m13.236507281s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.219631561 (375h7m13.259204681s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.223422469 (375h7m13.262995589s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.246388691 (375h7m13.285961821s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.662271802 (375h7m13.701844932s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: connect: no route to host; manager closed: conn closed: op Sep 22 04:35:37.981624261 (375h7m14.021197391s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033379768 (375h7m40.072952898s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033463777 (375h7m40.073036907s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033711309 (375h7m40.073284439s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:36:04.033742338 (375h7m40.073315468s) [seg_pos={0 0}, upload=329713]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:36:04.285892601 (375h7m40.325465731s) [upload=329713]: Done uploading segments... Sep 22 04:36:04.285932256 (375h7m40.325505386s) [upload=329713]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.143458403s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.......X.X.X....X......XXXXXXX.X......X.X.X..X.......X.XX.] duration: 6.032138409s FAIL (2025-09-22 04:37:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.032167455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 5.878021301s FAIL (2025-09-22 04:37:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.878053803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XX.......X.X.X....X......XXXXXXX.X......X.X.X..X.......X.XX] duration: 30.00048382s FAIL (2025-09-22 04:37:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000528394s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X.........XX..XX..X.X....XX........X.....X...X......X.X...] duration: 6.046380051s FAIL (2025-09-22 04:37:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:35:34.758938037 (375h7m10.798511177s) [upload=329713]: Starting upload Sep 22 04:35:34.759019573 (375h7m10.798592703s) [upload=329713]: Done starting upload Sep 22 04:35:34.759083996 (375h7m10.798657116s) [upload=329713]: Uploading segments... Sep 22 04:35:34.759305898 (375h7m10.798879028s) [upload=329713]: Got next segment. Inline: false Sep 22 04:35:34.759347898 (375h7m10.798921028s) [upload=329713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:35:34.759354610 (375h7m10.79892777s) [upload=329713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:35:35.037917458 (375h7m11.077490588s) [seg_pos={0 0}, upload=329713]: Begin upload segment... Sep 22 04:35:35.037974858 (375h7m11.077547988s) [seg_pos={0 0}, upload=329713]: Done begin upload segment. Sep 22 04:35:35.037993022 (375h7m11.077566142s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038009483 (375h7m11.077582614s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038066441 (375h7m11.077639571s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038432619 (375h7m11.078005759s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038583556 (375h7m11.078156686s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038940667 (375h7m11.078513787s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039287507 (375h7m11.078860627s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039682158 (375h7m11.079255279s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040009171 (375h7m11.079582301s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040491379 (375h7m11.080064509s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040750273 (375h7m11.080323393s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041190100 (375h7m11.08076322s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041528865 (375h7m11.081101995s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041913267 (375h7m11.081486397s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043178876 (375h7m11.082752016s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043813055 (375h7m11.083386185s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:36.831646663 (375h7m12.871219793s) [upload=329713]: Next returned nil segment Sep 22 04:35:36.831673043 (375h7m12.871246173s) [upload=329713]: Waiting for error group managing segments... Sep 22 04:35:36.958845684 (375h7m12.998418804s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:35:37.139170497 (375h7m13.178743617s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.140993289 (375h7m13.180566419s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.141524941 (375h7m13.181098071s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.142292071 (375h7m13.181865202s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.168995379 (375h7m13.208568509s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.183756498 (375h7m13.223329618s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.196934151 (375h7m13.236507281s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.219631561 (375h7m13.259204681s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.223422469 (375h7m13.262995589s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.246388691 (375h7m13.285961821s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.662271802 (375h7m13.701844932s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: connect: no route to host; manager closed: conn closed: op Sep 22 04:35:37.981624261 (375h7m14.021197391s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033379768 (375h7m40.072952898s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033463777 (375h7m40.073036907s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033711309 (375h7m40.073284439s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:36:04.033742338 (375h7m40.073315468s) [seg_pos={0 0}, upload=329713]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:36:04.285892601 (375h7m40.325465731s) [upload=329713]: Done uploading segments... Sep 22 04:36:04.285932256 (375h7m40.325505386s) [upload=329713]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.046421018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 6.178494059s FAIL (2025-09-22 04:37:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.178533583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 5.862708701s FAIL (2025-09-22 04:36:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.862753618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X.........XX..XX..X.X....XX........X.....X...X......X.X..] duration: 6.008157675s FAIL (2025-09-22 04:36:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:35:34.758938037 (375h7m10.798511177s) [upload=329713]: Starting upload Sep 22 04:35:34.759019573 (375h7m10.798592703s) [upload=329713]: Done starting upload Sep 22 04:35:34.759083996 (375h7m10.798657116s) [upload=329713]: Uploading segments... Sep 22 04:35:34.759305898 (375h7m10.798879028s) [upload=329713]: Got next segment. Inline: false Sep 22 04:35:34.759347898 (375h7m10.798921028s) [upload=329713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:35:34.759354610 (375h7m10.79892777s) [upload=329713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:35:35.037917458 (375h7m11.077490588s) [seg_pos={0 0}, upload=329713]: Begin upload segment... Sep 22 04:35:35.037974858 (375h7m11.077547988s) [seg_pos={0 0}, upload=329713]: Done begin upload segment. Sep 22 04:35:35.037993022 (375h7m11.077566142s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038009483 (375h7m11.077582614s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038066441 (375h7m11.077639571s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038432619 (375h7m11.078005759s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038583556 (375h7m11.078156686s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038940667 (375h7m11.078513787s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039287507 (375h7m11.078860627s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039682158 (375h7m11.079255279s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040009171 (375h7m11.079582301s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040491379 (375h7m11.080064509s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040750273 (375h7m11.080323393s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041190100 (375h7m11.08076322s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041528865 (375h7m11.081101995s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041913267 (375h7m11.081486397s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043178876 (375h7m11.082752016s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043813055 (375h7m11.083386185s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:36.831646663 (375h7m12.871219793s) [upload=329713]: Next returned nil segment Sep 22 04:35:36.831673043 (375h7m12.871246173s) [upload=329713]: Waiting for error group managing segments... Sep 22 04:35:36.958845684 (375h7m12.998418804s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:35:37.139170497 (375h7m13.178743617s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.140993289 (375h7m13.180566419s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.141524941 (375h7m13.181098071s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.142292071 (375h7m13.181865202s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.168995379 (375h7m13.208568509s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.183756498 (375h7m13.223329618s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.196934151 (375h7m13.236507281s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.219631561 (375h7m13.259204681s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.223422469 (375h7m13.262995589s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.246388691 (375h7m13.285961821s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.662271802 (375h7m13.701844932s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: connect: no route to host; manager closed: conn closed: op Sep 22 04:35:37.981624261 (375h7m14.021197391s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033379768 (375h7m40.072952898s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033463777 (375h7m40.073036907s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033711309 (375h7m40.073284439s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:36:04.033742338 (375h7m40.073315468s) [seg_pos={0 0}, upload=329713]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:36:04.285892601 (375h7m40.325465731s) [upload=329713]: Done uploading segments... Sep 22 04:36:04.285932256 (375h7m40.325505386s) [upload=329713]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.008191059s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X.......X.X] duration: 30.000991445s FAIL (2025-09-22 04:36:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001019848s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.X.........XX..XX..X.X....XX........X.....X...X......X.X.] duration: 6.311350515s FAIL (2025-09-22 04:36:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:35:34.758938037 (375h7m10.798511177s) [upload=329713]: Starting upload Sep 22 04:35:34.759019573 (375h7m10.798592703s) [upload=329713]: Done starting upload Sep 22 04:35:34.759083996 (375h7m10.798657116s) [upload=329713]: Uploading segments... Sep 22 04:35:34.759305898 (375h7m10.798879028s) [upload=329713]: Got next segment. Inline: false Sep 22 04:35:34.759347898 (375h7m10.798921028s) [upload=329713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:35:34.759354610 (375h7m10.79892777s) [upload=329713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:35:35.037917458 (375h7m11.077490588s) [seg_pos={0 0}, upload=329713]: Begin upload segment... Sep 22 04:35:35.037974858 (375h7m11.077547988s) [seg_pos={0 0}, upload=329713]: Done begin upload segment. Sep 22 04:35:35.037993022 (375h7m11.077566142s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038009483 (375h7m11.077582614s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038066441 (375h7m11.077639571s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038432619 (375h7m11.078005759s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038583556 (375h7m11.078156686s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038940667 (375h7m11.078513787s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039287507 (375h7m11.078860627s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039682158 (375h7m11.079255279s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040009171 (375h7m11.079582301s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040491379 (375h7m11.080064509s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040750273 (375h7m11.080323393s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041190100 (375h7m11.08076322s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041528865 (375h7m11.081101995s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041913267 (375h7m11.081486397s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043178876 (375h7m11.082752016s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043813055 (375h7m11.083386185s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:36.831646663 (375h7m12.871219793s) [upload=329713]: Next returned nil segment Sep 22 04:35:36.831673043 (375h7m12.871246173s) [upload=329713]: Waiting for error group managing segments... Sep 22 04:35:36.958845684 (375h7m12.998418804s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:35:37.139170497 (375h7m13.178743617s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.140993289 (375h7m13.180566419s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.141524941 (375h7m13.181098071s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.142292071 (375h7m13.181865202s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.168995379 (375h7m13.208568509s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.183756498 (375h7m13.223329618s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.196934151 (375h7m13.236507281s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.219631561 (375h7m13.259204681s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.223422469 (375h7m13.262995589s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.246388691 (375h7m13.285961821s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.662271802 (375h7m13.701844932s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: connect: no route to host; manager closed: conn closed: op Sep 22 04:35:37.981624261 (375h7m14.021197391s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033379768 (375h7m40.072952898s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033463777 (375h7m40.073036907s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033711309 (375h7m40.073284439s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:36:04.033742338 (375h7m40.073315468s) [seg_pos={0 0}, upload=329713]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:36:04.285892601 (375h7m40.325465731s) [upload=329713]: Done uploading segments... Sep 22 04:36:04.285932256 (375h7m40.325505386s) [upload=329713]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.311371775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X.X.........XX..XX..X.X....XX........X.....X...X......X.X] duration: 30.255085074s FAIL (2025-09-22 04:36:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:35:34.758938037 (375h7m10.798511177s) [upload=329713]: Starting upload Sep 22 04:35:34.759019573 (375h7m10.798592703s) [upload=329713]: Done starting upload Sep 22 04:35:34.759083996 (375h7m10.798657116s) [upload=329713]: Uploading segments... Sep 22 04:35:34.759305898 (375h7m10.798879028s) [upload=329713]: Got next segment. Inline: false Sep 22 04:35:34.759347898 (375h7m10.798921028s) [upload=329713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:35:34.759354610 (375h7m10.79892777s) [upload=329713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:35:35.037917458 (375h7m11.077490588s) [seg_pos={0 0}, upload=329713]: Begin upload segment... Sep 22 04:35:35.037974858 (375h7m11.077547988s) [seg_pos={0 0}, upload=329713]: Done begin upload segment. Sep 22 04:35:35.037993022 (375h7m11.077566142s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038009483 (375h7m11.077582614s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038066441 (375h7m11.077639571s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038432619 (375h7m11.078005759s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038583556 (375h7m11.078156686s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.038940667 (375h7m11.078513787s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039287507 (375h7m11.078860627s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.039682158 (375h7m11.079255279s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040009171 (375h7m11.079582301s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040491379 (375h7m11.080064509s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.040750273 (375h7m11.080323393s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041190100 (375h7m11.08076322s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041528865 (375h7m11.081101995s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.041913267 (375h7m11.081486397s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043178876 (375h7m11.082752016s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:35.043813055 (375h7m11.083386185s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Uploading piece... Sep 22 04:35:36.831646663 (375h7m12.871219793s) [upload=329713]: Next returned nil segment Sep 22 04:35:36.831673043 (375h7m12.871246173s) [upload=329713]: Waiting for error group managing segments... Sep 22 04:35:36.958845684 (375h7m12.998418804s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZWNYLKZXCLAQDZQXP5WC27YP6RXOII2WMTVB3JKBLWA47UG7TA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:35:37.139170497 (375h7m13.178743617s) [noise=true , address=45.91.116.12:4227 , piece_id=GPHRXZ5OLL4QXQNPZ5SMFQ3GNIYMRQBE2FR3NOWBQHNJR4JXRQKA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.140993289 (375h7m13.180566419s) [noise=true , address=45.91.116.4:4938 , piece_id=5OHVIRKCCBKYAVIVO3N756O7LYX7U3AA3ZTNM45EJH757N2MFJQQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.141524941 (375h7m13.181098071s) [noise=true , address=45.91.116.11:4117 , piece_id=TCOXDDUJSJ2DOI7QEIF4WZ5A57RA7XG7WZDXSU7CWBVZXNEWLTFA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.142292071 (375h7m13.181865202s) [noise=true , address=45.91.116.4:4939 , piece_id=I3XBYDH5VZETDVQUA4SJIJOTGXC2YQCOQFG6CVFKIQCY2IVX26SA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.168995379 (375h7m13.208568509s) [noise=true , address=77.90.31.16:4735 , piece_id=57V7DIVEQNYP2ABPWSJLZYQGKDFMFK3OQQ2YLYACKAFQBARZGV4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.183756498 (375h7m13.223329618s) [noise=true , address=45.91.116.13:4153 , piece_id=ZUIWKUCP6GVPS7SFNPWFKXJGDCND3OIKIX77KSR4SD2JAADH4E7Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.196934151 (375h7m13.236507281s) [noise=true , address=77.90.31.12:4592 , piece_id=L6WKOBF2BDZ3XMLCESIP5PNTQ724CEUUFBBNFGAIKW4LD4MACDVA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.219631561 (375h7m13.259204681s) [noise=true , address=77.90.31.13:4628 , piece_id=Q4OV2NPRP46UISL5LJ3UGGKFSBDU55UAV6IX7M6WWJWYG6RS5IYQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.223422469 (375h7m13.262995589s) [noise=true , address=45.91.116.6:4975 , piece_id=DCCEU6IWMOQTK2SUUZGZO6ASPNC56JE7LIZXE5BMNNH3453YXS6A, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.246388691 (375h7m13.285961821s) [noise=true , address=77.90.31.18:4806 , piece_id=7DFF2XUAT6I54V6QIQPZ3CLZ3RZMOT24BMPHTYL45FZVAWTSS3SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: <nil> Sep 22 04:35:37.662271802 (375h7m13.701844932s) [noise=true , address=145.79.227.5:4221 , piece_id=NEM6HXUS65I6XIY2URTYIMD6PMNTBN3NHSCFJKXQ7GAOXL6TEODQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: connect: no route to host; manager closed: conn closed: op Sep 22 04:35:37.981624261 (375h7m14.021197391s) [noise=true , address=145.79.227.16:4912 , piece_id=GXK3TELAOMCQ353TVYO3AHUOAP6HFAWWOZPLMCOUZMKRVRPY6B4Q, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033379768 (375h7m40.072952898s) [noise=true , address=145.79.227.18:4005 , piece_id=LITHZ37ZMY2ICILWLFEZ6NW6RH5ECLVTQEIWDNJIWLM2E44GBXKQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033463777 (375h7m40.073036907s) [noise=true , address=145.79.227.7:4293 , piece_id=LB4EPMPWN3NQI6WG7HP5B7EZTSZQ5NVGRXILAFWJ6VYVDQE3L6GA, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:04.033711309 (375h7m40.073284439s) [noise=true , address=145.79.227.17:4969 , piece_id=EMJ3VCYYA7EMEKVJV2BNJ57CAUEZ3O7L5DRKVGC4AK7DBRT256SQ, seg_pos={0 0}, upload=329713]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:36:04.033742338 (375h7m40.073315468s) [seg_pos={0 0}, upload=329713]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:36:04.285892601 (375h7m40.325465731s) [upload=329713]: Done uploading segments... Sep 22 04:36:04.285932256 (375h7m40.325505386s) [upload=329713]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.25511463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 5.92469602s FAIL (2025-09-22 04:35:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.924726227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X.......X.] duration: 6.129342076s FAIL (2025-09-22 04:35:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.129367784s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X.......X] duration: 30.001208972s FAIL (2025-09-22 04:35:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001241884s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 6.147507098s FAIL (2025-09-22 04:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.14754451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.X.X.........XX..XX..X.X....XX........X.....X...X......X.] duration: 6.419482518s FAIL (2025-09-22 04:34:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:34:00.716243697 (375h5m36.755816827s) [upload=329692]: Starting upload Sep 22 04:34:00.716398401 (375h5m36.755971531s) [upload=329692]: Done starting upload Sep 22 04:34:00.716540742 (375h5m36.756113862s) [upload=329692]: Uploading segments... Sep 22 04:34:00.716670409 (375h5m36.756243529s) [upload=329692]: Got next segment. Inline: false Sep 22 04:34:00.716768687 (375h5m36.756341817s) [upload=329692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:34:00.716775018 (375h5m36.756348139s) [upload=329692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:34:00.967408161 (375h5m37.006981301s) [seg_pos={0 0}, upload=329692]: Begin upload segment... Sep 22 04:34:00.967464479 (375h5m37.007037619s) [seg_pos={0 0}, upload=329692]: Done begin upload segment. Sep 22 04:34:00.967495769 (375h5m37.007068899s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWZ5NSWAXRDUOAVPX7UKYU2R4TUS36WUJ6SBXVQLCUL7LEEBWRGA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.967579718 (375h5m37.007152838s) [noise=true , address=77.90.31.11:4553 , piece_id=3I3X5G6MJPIWNOSWECJGQOEG7F2XTBN4F47FVJ5R5CCITNCWJ24A, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.967987755 (375h5m37.007560875s) [noise=true , address=145.79.227.17:4971 , piece_id=6SJ3HXYPTSXEQZHT7Q644GN4U26ETBBMYAQFL35JFE2L22K6US4A, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968133362 (375h5m37.007706493s) [noise=true , address=77.90.31.15:4698 , piece_id=HZE5DZEZN2AJE3SSJJNEVOE4LKH73SYBL4MOGRM66ODL2X46P46A, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968418696 (375h5m37.007991826s) [noise=true , address=77.90.31.17:4770 , piece_id=MFTESTVMOSLO6H6SI4ENZCENWDJMXDMHT6FKCXCZAKUPTBGHIFHQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968540167 (375h5m37.008113287s) [noise=true , address=45.91.116.6:4973 , piece_id=A2SYW3PFWBLKYWJTN2RJKKQSUBGNRT3WKAQPSRSEPNJG4B66CDHA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968778711 (375h5m37.008351841s) [noise=true , address=77.90.31.16:4734 , piece_id=QXNZFRI2TYB6DLMPTJO7UMW7UE273BPF6NSF67NBES6IYCZAMVLQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968795844 (375h5m37.008368964s) [noise=true , address=45.91.116.9:4261 , piece_id=TACMKR3M67JOZN35AGHCTLWGZ7V5WGLR4U6JZTFMNICFTGXB6FQQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969122335 (375h5m37.008695465s) [noise=true , address=145.79.227.16:4910 , piece_id=BVUF6HY7LQZIYUL5L4U7GI7INRQUIQ54UMH46QSYOPOSAKY3TF4A, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969136282 (375h5m37.008709412s) [noise=true , address=45.91.116.13:4153 , piece_id=GK2RLVMUQIDOMBAVHXAGT4OP4LZ43SQABO4UD6PFOY33PRZ4MAKQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969400485 (375h5m37.008973615s) [noise=true , address=145.79.227.13:4804 , piece_id=NC7R5DXLFY62LKJ6DMLPSP4ECGMJ3LZUV5GZMBHZNX6MCWVZAYYA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969542736 (375h5m37.009115866s) [noise=true , address=45.91.116.5:4297 , piece_id=MKLITNEJMSREPKIXNTZ5LPDEKHOAJXYMKF4WUQ5QLVVQUP5ST7SA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969662865 (375h5m37.009235995s) [noise=true , address=45.91.116.10:4081 , piece_id=LIODF733QKAXQYMJRN45T2QJF52SMS2VSEOAUI7ZRFZDBEWIFCOA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969996490 (375h5m37.00956962s) [noise=true , address=45.91.116.14:4190 , piece_id=XLRXTR5I6PQ35EJYROQZZNFKVEC74JMIMM6RITNSNSSZ6FMD4VQQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.970347518 (375h5m37.009920648s) [noise=true , address=77.90.31.19:4842 , piece_id=3G3PX44I3KKHHCIS5QJ2RRMYBNND42TVJ6EA73OLJYVP7FBRPTVA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.972894298 (375h5m37.012467418s) [noise=true , address=145.79.227.14:4839 , piece_id=PB2KR6QYAWQ5TZK5CSHWBCAU3XFEJECEKMAXFXPUICRRKN545PCQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:02.718279957 (375h5m38.757853097s) [upload=329692]: Next returned nil segment Sep 22 04:34:02.718305866 (375h5m38.757878996s) [upload=329692]: Waiting for error group managing segments... Sep 22 04:34:03.066411526 (375h5m39.105984667s) [noise=true , address=45.91.116.6:4973 , piece_id=A2SYW3PFWBLKYWJTN2RJKKQSUBGNRT3WKAQPSRSEPNJG4B66CDHA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.073219990 (375h5m39.11279311s) [noise=true , address=45.91.116.9:4261 , piece_id=TACMKR3M67JOZN35AGHCTLWGZ7V5WGLR4U6JZTFMNICFTGXB6FQQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.073648798 (375h5m39.113221928s) [noise=true , address=77.90.31.16:4734 , piece_id=QXNZFRI2TYB6DLMPTJO7UMW7UE273BPF6NSF67NBES6IYCZAMVLQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.073874607 (375h5m39.113447737s) [noise=true , address=45.91.116.10:4081 , piece_id=LIODF733QKAXQYMJRN45T2QJF52SMS2VSEOAUI7ZRFZDBEWIFCOA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.074478937 (375h5m39.114052068s) [noise=true , address=77.90.31.19:4842 , piece_id=3G3PX44I3KKHHCIS5QJ2RRMYBNND42TVJ6EA73OLJYVP7FBRPTVA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.116702620 (375h5m39.15627575s) [noise=true , address=45.91.116.5:4297 , piece_id=MKLITNEJMSREPKIXNTZ5LPDEKHOAJXYMKF4WUQ5QLVVQUP5ST7SA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.121113860 (375h5m39.16068699s) [noise=true , address=77.90.31.15:4698 , piece_id=HZE5DZEZN2AJE3SSJJNEVOE4LKH73SYBL4MOGRM66ODL2X46P46A, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.141598286 (375h5m39.181171416s) [noise=true , address=45.91.116.14:4190 , piece_id=XLRXTR5I6PQ35EJYROQZZNFKVEC74JMIMM6RITNSNSSZ6FMD4VQQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.147114771 (375h5m39.186687911s) [noise=true , address=45.91.116.13:4153 , piece_id=GK2RLVMUQIDOMBAVHXAGT4OP4LZ43SQABO4UD6PFOY33PRZ4MAKQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.157998315 (375h5m39.197571445s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWZ5NSWAXRDUOAVPX7UKYU2R4TUS36WUJ6SBXVQLCUL7LEEBWRGA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.166842475 (375h5m39.206415605s) [noise=true , address=77.90.31.17:4770 , piece_id=MFTESTVMOSLO6H6SI4ENZCENWDJMXDMHT6FKCXCZAKUPTBGHIFHQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.171943018 (375h5m39.211516158s) [noise=true , address=77.90.31.11:4553 , piece_id=3I3X5G6MJPIWNOSWECJGQOEG7F2XTBN4F47FVJ5R5CCITNCWJ24A, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:05.790189421 (375h5m41.829762551s) [noise=true , address=145.79.227.16:4910 , piece_id=BVUF6HY7LQZIYUL5L4U7GI7INRQUIQ54UMH46QSYOPOSAKY3TF4A, seg_pos={0 0}, upload=329692]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op Sep 22 04:34:30.013034877 (375h6m6.052608007s) [noise=true , address=145.79.227.14:4839 , piece_id=PB2KR6QYAWQ5TZK5CSHWBCAU3XFEJECEKMAXFXPUICRRKN545PCQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:34:30.013179763 (375h6m6.052752883s) [noise=true , address=145.79.227.17:4971 , piece_id=6SJ3HXYPTSXEQZHT7Q644GN4U26ETBBMYAQFL35JFE2L22K6US4A, seg_pos={0 0}, upload=329692]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:34:30.013309821 (375h6m6.052882951s) [noise=true , address=145.79.227.13:4804 , piece_id=NC7R5DXLFY62LKJ6DMLPSP4ECGMJ3LZUV5GZMBHZNX6MCWVZAYYA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: rpc: dial tcp 145.79.227.13:4804: i/o timeout; context deadline exceeded Sep 22 04:34:30.013357782 (375h6m6.052930902s) [seg_pos={0 0}, upload=329692]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:34:30.257452615 (375h6m6.297025735s) [upload=329692]: Done uploading segments... Sep 22 04:34:30.257481429 (375h6m6.297054549s) [upload=329692]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.419508978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X.X.X.........XX..XX..X.X....XX........X.....X...X......X] duration: 30.247375185s FAIL (2025-09-22 04:34:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:34:00.716243697 (375h5m36.755816827s) [upload=329692]: Starting upload Sep 22 04:34:00.716398401 (375h5m36.755971531s) [upload=329692]: Done starting upload Sep 22 04:34:00.716540742 (375h5m36.756113862s) [upload=329692]: Uploading segments... Sep 22 04:34:00.716670409 (375h5m36.756243529s) [upload=329692]: Got next segment. Inline: false Sep 22 04:34:00.716768687 (375h5m36.756341817s) [upload=329692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:34:00.716775018 (375h5m36.756348139s) [upload=329692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:34:00.967408161 (375h5m37.006981301s) [seg_pos={0 0}, upload=329692]: Begin upload segment... Sep 22 04:34:00.967464479 (375h5m37.007037619s) [seg_pos={0 0}, upload=329692]: Done begin upload segment. Sep 22 04:34:00.967495769 (375h5m37.007068899s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWZ5NSWAXRDUOAVPX7UKYU2R4TUS36WUJ6SBXVQLCUL7LEEBWRGA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.967579718 (375h5m37.007152838s) [noise=true , address=77.90.31.11:4553 , piece_id=3I3X5G6MJPIWNOSWECJGQOEG7F2XTBN4F47FVJ5R5CCITNCWJ24A, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.967987755 (375h5m37.007560875s) [noise=true , address=145.79.227.17:4971 , piece_id=6SJ3HXYPTSXEQZHT7Q644GN4U26ETBBMYAQFL35JFE2L22K6US4A, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968133362 (375h5m37.007706493s) [noise=true , address=77.90.31.15:4698 , piece_id=HZE5DZEZN2AJE3SSJJNEVOE4LKH73SYBL4MOGRM66ODL2X46P46A, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968418696 (375h5m37.007991826s) [noise=true , address=77.90.31.17:4770 , piece_id=MFTESTVMOSLO6H6SI4ENZCENWDJMXDMHT6FKCXCZAKUPTBGHIFHQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968540167 (375h5m37.008113287s) [noise=true , address=45.91.116.6:4973 , piece_id=A2SYW3PFWBLKYWJTN2RJKKQSUBGNRT3WKAQPSRSEPNJG4B66CDHA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968778711 (375h5m37.008351841s) [noise=true , address=77.90.31.16:4734 , piece_id=QXNZFRI2TYB6DLMPTJO7UMW7UE273BPF6NSF67NBES6IYCZAMVLQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.968795844 (375h5m37.008368964s) [noise=true , address=45.91.116.9:4261 , piece_id=TACMKR3M67JOZN35AGHCTLWGZ7V5WGLR4U6JZTFMNICFTGXB6FQQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969122335 (375h5m37.008695465s) [noise=true , address=145.79.227.16:4910 , piece_id=BVUF6HY7LQZIYUL5L4U7GI7INRQUIQ54UMH46QSYOPOSAKY3TF4A, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969136282 (375h5m37.008709412s) [noise=true , address=45.91.116.13:4153 , piece_id=GK2RLVMUQIDOMBAVHXAGT4OP4LZ43SQABO4UD6PFOY33PRZ4MAKQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969400485 (375h5m37.008973615s) [noise=true , address=145.79.227.13:4804 , piece_id=NC7R5DXLFY62LKJ6DMLPSP4ECGMJ3LZUV5GZMBHZNX6MCWVZAYYA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969542736 (375h5m37.009115866s) [noise=true , address=45.91.116.5:4297 , piece_id=MKLITNEJMSREPKIXNTZ5LPDEKHOAJXYMKF4WUQ5QLVVQUP5ST7SA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969662865 (375h5m37.009235995s) [noise=true , address=45.91.116.10:4081 , piece_id=LIODF733QKAXQYMJRN45T2QJF52SMS2VSEOAUI7ZRFZDBEWIFCOA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.969996490 (375h5m37.00956962s) [noise=true , address=45.91.116.14:4190 , piece_id=XLRXTR5I6PQ35EJYROQZZNFKVEC74JMIMM6RITNSNSSZ6FMD4VQQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.970347518 (375h5m37.009920648s) [noise=true , address=77.90.31.19:4842 , piece_id=3G3PX44I3KKHHCIS5QJ2RRMYBNND42TVJ6EA73OLJYVP7FBRPTVA, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:00.972894298 (375h5m37.012467418s) [noise=true , address=145.79.227.14:4839 , piece_id=PB2KR6QYAWQ5TZK5CSHWBCAU3XFEJECEKMAXFXPUICRRKN545PCQ, seg_pos={0 0}, upload=329692]: Uploading piece... Sep 22 04:34:02.718279957 (375h5m38.757853097s) [upload=329692]: Next returned nil segment Sep 22 04:34:02.718305866 (375h5m38.757878996s) [upload=329692]: Waiting for error group managing segments... Sep 22 04:34:03.066411526 (375h5m39.105984667s) [noise=true , address=45.91.116.6:4973 , piece_id=A2SYW3PFWBLKYWJTN2RJKKQSUBGNRT3WKAQPSRSEPNJG4B66CDHA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.073219990 (375h5m39.11279311s) [noise=true , address=45.91.116.9:4261 , piece_id=TACMKR3M67JOZN35AGHCTLWGZ7V5WGLR4U6JZTFMNICFTGXB6FQQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.073648798 (375h5m39.113221928s) [noise=true , address=77.90.31.16:4734 , piece_id=QXNZFRI2TYB6DLMPTJO7UMW7UE273BPF6NSF67NBES6IYCZAMVLQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.073874607 (375h5m39.113447737s) [noise=true , address=45.91.116.10:4081 , piece_id=LIODF733QKAXQYMJRN45T2QJF52SMS2VSEOAUI7ZRFZDBEWIFCOA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.074478937 (375h5m39.114052068s) [noise=true , address=77.90.31.19:4842 , piece_id=3G3PX44I3KKHHCIS5QJ2RRMYBNND42TVJ6EA73OLJYVP7FBRPTVA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.116702620 (375h5m39.15627575s) [noise=true , address=45.91.116.5:4297 , piece_id=MKLITNEJMSREPKIXNTZ5LPDEKHOAJXYMKF4WUQ5QLVVQUP5ST7SA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.121113860 (375h5m39.16068699s) [noise=true , address=77.90.31.15:4698 , piece_id=HZE5DZEZN2AJE3SSJJNEVOE4LKH73SYBL4MOGRM66ODL2X46P46A, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.141598286 (375h5m39.181171416s) [noise=true , address=45.91.116.14:4190 , piece_id=XLRXTR5I6PQ35EJYROQZZNFKVEC74JMIMM6RITNSNSSZ6FMD4VQQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.147114771 (375h5m39.186687911s) [noise=true , address=45.91.116.13:4153 , piece_id=GK2RLVMUQIDOMBAVHXAGT4OP4LZ43SQABO4UD6PFOY33PRZ4MAKQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.157998315 (375h5m39.197571445s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWZ5NSWAXRDUOAVPX7UKYU2R4TUS36WUJ6SBXVQLCUL7LEEBWRGA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.166842475 (375h5m39.206415605s) [noise=true , address=77.90.31.17:4770 , piece_id=MFTESTVMOSLO6H6SI4ENZCENWDJMXDMHT6FKCXCZAKUPTBGHIFHQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:03.171943018 (375h5m39.211516158s) [noise=true , address=77.90.31.11:4553 , piece_id=3I3X5G6MJPIWNOSWECJGQOEG7F2XTBN4F47FVJ5R5CCITNCWJ24A, seg_pos={0 0}, upload=329692]: Done uploading piece. err: <nil> Sep 22 04:34:05.790189421 (375h5m41.829762551s) [noise=true , address=145.79.227.16:4910 , piece_id=BVUF6HY7LQZIYUL5L4U7GI7INRQUIQ54UMH46QSYOPOSAKY3TF4A, seg_pos={0 0}, upload=329692]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op Sep 22 04:34:30.013034877 (375h6m6.052608007s) [noise=true , address=145.79.227.14:4839 , piece_id=PB2KR6QYAWQ5TZK5CSHWBCAU3XFEJECEKMAXFXPUICRRKN545PCQ, seg_pos={0 0}, upload=329692]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:34:30.013179763 (375h6m6.052752883s) [noise=true , address=145.79.227.17:4971 , piece_id=6SJ3HXYPTSXEQZHT7Q644GN4U26ETBBMYAQFL35JFE2L22K6US4A, seg_pos={0 0}, upload=329692]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:34:30.013309821 (375h6m6.052882951s) [noise=true , address=145.79.227.13:4804 , piece_id=NC7R5DXLFY62LKJ6DMLPSP4ECGMJ3LZUV5GZMBHZNX6MCWVZAYYA, seg_pos={0 0}, upload=329692]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: rpc: dial tcp 145.79.227.13:4804: i/o timeout; context deadline exceeded Sep 22 04:34:30.013357782 (375h6m6.052930902s) [seg_pos={0 0}, upload=329692]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:34:30.257452615 (375h6m6.297025735s) [upload=329692]: Done uploading segments... Sep 22 04:34:30.257481429 (375h6m6.297054549s) [upload=329692]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.247407406s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 6.131080011s FAIL (2025-09-22 04:34:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.131160666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X.......] duration: 6.351260197s FAIL (2025-09-22 04:34:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.351284934s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...X.X.X.........XX..XX..X.X....XX........X.....X...X......] duration: 6.157914421s FAIL (2025-09-22 04:33:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:29:06.120000576 (375h0m42.159573706s) [upload=329615]: Starting upload Sep 22 04:29:06.120076982 (375h0m42.159650102s) [upload=329615]: Done starting upload Sep 22 04:29:06.120198202 (375h0m42.159771333s) [upload=329615]: Uploading segments... Sep 22 04:29:06.120374228 (375h0m42.159947368s) [upload=329615]: Got next segment. Inline: false Sep 22 04:29:06.120423472 (375h0m42.159996602s) [upload=329615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:29:06.120429143 (375h0m42.160002263s) [upload=329615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:29:06.615739730 (375h0m42.65531285s) [seg_pos={0 0}, upload=329615]: Begin upload segment... Sep 22 04:29:06.615791668 (375h0m42.655364808s) [seg_pos={0 0}, upload=329615]: Done begin upload segment. Sep 22 04:29:06.615819902 (375h0m42.655393032s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616019633 (375h0m42.655592763s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616230955 (375h0m42.655804085s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616691882 (375h0m42.656265003s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616840605 (375h0m42.656413735s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616945585 (375h0m42.656518715s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617188929 (375h0m42.656762049s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617355596 (375h0m42.656928726s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617450227 (375h0m42.657023357s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617842974 (375h0m42.657416104s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617946411 (375h0m42.657519541s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618175438 (375h0m42.657748568s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618493012 (375h0m42.658066142s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618513361 (375h0m42.658086481s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.619067175 (375h0m42.658640305s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.622292537 (375h0m42.661865667s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:08.380779543 (375h0m44.420352673s) [upload=329615]: Next returned nil segment Sep 22 04:29:08.380797839 (375h0m44.420370969s) [upload=329615]: Waiting for error group managing segments... Sep 22 04:29:08.674700405 (375h0m44.714273525s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.709729657 (375h0m44.749302798s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.721175112 (375h0m44.760748232s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.728378538 (375h0m44.767951669s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.729967575 (375h0m44.769540705s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.736253273 (375h0m44.775826403s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.747689690 (375h0m44.787262821s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.768535595 (375h0m44.808108715s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.771670174 (375h0m44.811243294s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.772745321 (375h0m44.812318451s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.780813184 (375h0m44.820386314s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.789877134 (375h0m44.829450264s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:35.394668981 (375h1m11.434242111s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394720769 (375h1m11.434293899s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394765194 (375h1m11.434338324s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: rpc: dial tcp 145.79.227.3:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394804739 (375h1m11.434377869s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394821110 (375h1m11.43439424s) [seg_pos={0 0}, upload=329615]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:29:35.636648055 (375h1m11.676221175s) [upload=329615]: Done uploading segments... Sep 22 04:29:35.636675467 (375h1m11.676248587s) [upload=329615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.157958044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 6.039157664s FAIL (2025-09-22 04:33:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.039196579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X......] duration: 6.216657868s FAIL (2025-09-22 04:33:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.216694828s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X...X.X.X.........XX..XX..X.X....XX........X.....X...X.....] duration: 6.050570267s FAIL (2025-09-22 04:32:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:29:06.120000576 (375h0m42.159573706s) [upload=329615]: Starting upload Sep 22 04:29:06.120076982 (375h0m42.159650102s) [upload=329615]: Done starting upload Sep 22 04:29:06.120198202 (375h0m42.159771333s) [upload=329615]: Uploading segments... Sep 22 04:29:06.120374228 (375h0m42.159947368s) [upload=329615]: Got next segment. Inline: false Sep 22 04:29:06.120423472 (375h0m42.159996602s) [upload=329615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:29:06.120429143 (375h0m42.160002263s) [upload=329615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:29:06.615739730 (375h0m42.65531285s) [seg_pos={0 0}, upload=329615]: Begin upload segment... Sep 22 04:29:06.615791668 (375h0m42.655364808s) [seg_pos={0 0}, upload=329615]: Done begin upload segment. Sep 22 04:29:06.615819902 (375h0m42.655393032s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616019633 (375h0m42.655592763s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616230955 (375h0m42.655804085s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616691882 (375h0m42.656265003s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616840605 (375h0m42.656413735s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616945585 (375h0m42.656518715s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617188929 (375h0m42.656762049s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617355596 (375h0m42.656928726s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617450227 (375h0m42.657023357s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617842974 (375h0m42.657416104s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617946411 (375h0m42.657519541s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618175438 (375h0m42.657748568s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618493012 (375h0m42.658066142s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618513361 (375h0m42.658086481s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.619067175 (375h0m42.658640305s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.622292537 (375h0m42.661865667s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:08.380779543 (375h0m44.420352673s) [upload=329615]: Next returned nil segment Sep 22 04:29:08.380797839 (375h0m44.420370969s) [upload=329615]: Waiting for error group managing segments... Sep 22 04:29:08.674700405 (375h0m44.714273525s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.709729657 (375h0m44.749302798s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.721175112 (375h0m44.760748232s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.728378538 (375h0m44.767951669s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.729967575 (375h0m44.769540705s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.736253273 (375h0m44.775826403s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.747689690 (375h0m44.787262821s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.768535595 (375h0m44.808108715s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.771670174 (375h0m44.811243294s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.772745321 (375h0m44.812318451s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.780813184 (375h0m44.820386314s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.789877134 (375h0m44.829450264s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:35.394668981 (375h1m11.434242111s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394720769 (375h1m11.434293899s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394765194 (375h1m11.434338324s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: rpc: dial tcp 145.79.227.3:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394804739 (375h1m11.434377869s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394821110 (375h1m11.43439424s) [seg_pos={0 0}, upload=329615]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:29:35.636648055 (375h1m11.676221175s) [upload=329615]: Done uploading segments... Sep 22 04:29:35.636675467 (375h1m11.676248587s) [upload=329615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.050604092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 6.018216516s FAIL (2025-09-22 04:32:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.018248757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X...X.X.X.........XX..XX..X.X....XX........X.....X...X....] duration: 6.071618945s FAIL (2025-09-22 04:32:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:29:06.120000576 (375h0m42.159573706s) [upload=329615]: Starting upload Sep 22 04:29:06.120076982 (375h0m42.159650102s) [upload=329615]: Done starting upload Sep 22 04:29:06.120198202 (375h0m42.159771333s) [upload=329615]: Uploading segments... Sep 22 04:29:06.120374228 (375h0m42.159947368s) [upload=329615]: Got next segment. Inline: false Sep 22 04:29:06.120423472 (375h0m42.159996602s) [upload=329615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:29:06.120429143 (375h0m42.160002263s) [upload=329615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:29:06.615739730 (375h0m42.65531285s) [seg_pos={0 0}, upload=329615]: Begin upload segment... Sep 22 04:29:06.615791668 (375h0m42.655364808s) [seg_pos={0 0}, upload=329615]: Done begin upload segment. Sep 22 04:29:06.615819902 (375h0m42.655393032s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616019633 (375h0m42.655592763s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616230955 (375h0m42.655804085s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616691882 (375h0m42.656265003s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616840605 (375h0m42.656413735s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616945585 (375h0m42.656518715s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617188929 (375h0m42.656762049s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617355596 (375h0m42.656928726s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617450227 (375h0m42.657023357s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617842974 (375h0m42.657416104s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617946411 (375h0m42.657519541s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618175438 (375h0m42.657748568s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618493012 (375h0m42.658066142s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618513361 (375h0m42.658086481s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.619067175 (375h0m42.658640305s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.622292537 (375h0m42.661865667s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:08.380779543 (375h0m44.420352673s) [upload=329615]: Next returned nil segment Sep 22 04:29:08.380797839 (375h0m44.420370969s) [upload=329615]: Waiting for error group managing segments... Sep 22 04:29:08.674700405 (375h0m44.714273525s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.709729657 (375h0m44.749302798s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.721175112 (375h0m44.760748232s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.728378538 (375h0m44.767951669s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.729967575 (375h0m44.769540705s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.736253273 (375h0m44.775826403s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.747689690 (375h0m44.787262821s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.768535595 (375h0m44.808108715s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.771670174 (375h0m44.811243294s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.772745321 (375h0m44.812318451s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.780813184 (375h0m44.820386314s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.789877134 (375h0m44.829450264s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:35.394668981 (375h1m11.434242111s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394720769 (375h1m11.434293899s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394765194 (375h1m11.434338324s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: rpc: dial tcp 145.79.227.3:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394804739 (375h1m11.434377869s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394821110 (375h1m11.43439424s) [seg_pos={0 0}, upload=329615]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:29:35.636648055 (375h1m11.676221175s) [upload=329615]: Done uploading segments... Sep 22 04:29:35.636675467 (375h1m11.676248587s) [upload=329615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.071655594s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.X..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X.....] duration: 6.161226287s FAIL (2025-09-22 04:32:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.161254251s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 5.976526935s FAIL (2025-09-22 04:32:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.976558576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X.X..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X....] duration: 5.93303101s FAIL (2025-09-22 04:31:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.933061397s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 5.975161393s FAIL (2025-09-22 04:31:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.975183735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X...X.X.X.........XX..XX..X.X....XX........X.....X...X...] duration: 6.076233313s FAIL (2025-09-22 04:31:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:29:06.120000576 (375h0m42.159573706s) [upload=329615]: Starting upload Sep 22 04:29:06.120076982 (375h0m42.159650102s) [upload=329615]: Done starting upload Sep 22 04:29:06.120198202 (375h0m42.159771333s) [upload=329615]: Uploading segments... Sep 22 04:29:06.120374228 (375h0m42.159947368s) [upload=329615]: Got next segment. Inline: false Sep 22 04:29:06.120423472 (375h0m42.159996602s) [upload=329615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:29:06.120429143 (375h0m42.160002263s) [upload=329615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:29:06.615739730 (375h0m42.65531285s) [seg_pos={0 0}, upload=329615]: Begin upload segment... Sep 22 04:29:06.615791668 (375h0m42.655364808s) [seg_pos={0 0}, upload=329615]: Done begin upload segment. Sep 22 04:29:06.615819902 (375h0m42.655393032s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616019633 (375h0m42.655592763s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616230955 (375h0m42.655804085s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616691882 (375h0m42.656265003s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616840605 (375h0m42.656413735s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616945585 (375h0m42.656518715s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617188929 (375h0m42.656762049s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617355596 (375h0m42.656928726s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617450227 (375h0m42.657023357s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617842974 (375h0m42.657416104s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617946411 (375h0m42.657519541s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618175438 (375h0m42.657748568s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618493012 (375h0m42.658066142s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618513361 (375h0m42.658086481s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.619067175 (375h0m42.658640305s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.622292537 (375h0m42.661865667s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:08.380779543 (375h0m44.420352673s) [upload=329615]: Next returned nil segment Sep 22 04:29:08.380797839 (375h0m44.420370969s) [upload=329615]: Waiting for error group managing segments... Sep 22 04:29:08.674700405 (375h0m44.714273525s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.709729657 (375h0m44.749302798s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.721175112 (375h0m44.760748232s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.728378538 (375h0m44.767951669s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.729967575 (375h0m44.769540705s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.736253273 (375h0m44.775826403s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.747689690 (375h0m44.787262821s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.768535595 (375h0m44.808108715s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.771670174 (375h0m44.811243294s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.772745321 (375h0m44.812318451s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.780813184 (375h0m44.820386314s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.789877134 (375h0m44.829450264s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:35.394668981 (375h1m11.434242111s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394720769 (375h1m11.434293899s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394765194 (375h1m11.434338324s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: rpc: dial tcp 145.79.227.3:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394804739 (375h1m11.434377869s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394821110 (375h1m11.43439424s) [seg_pos={0 0}, upload=329615]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:29:35.636648055 (375h1m11.676221175s) [upload=329615]: Done uploading segments... Sep 22 04:29:35.636675467 (375h1m11.676248587s) [upload=329615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.07626388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X...X.X.X.........XX..XX..X.X....XX........X.....X...X..] duration: 6.420026885s FAIL (2025-09-22 04:31:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:29:06.120000576 (375h0m42.159573706s) [upload=329615]: Starting upload Sep 22 04:29:06.120076982 (375h0m42.159650102s) [upload=329615]: Done starting upload Sep 22 04:29:06.120198202 (375h0m42.159771333s) [upload=329615]: Uploading segments... Sep 22 04:29:06.120374228 (375h0m42.159947368s) [upload=329615]: Got next segment. Inline: false Sep 22 04:29:06.120423472 (375h0m42.159996602s) [upload=329615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:29:06.120429143 (375h0m42.160002263s) [upload=329615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:29:06.615739730 (375h0m42.65531285s) [seg_pos={0 0}, upload=329615]: Begin upload segment... Sep 22 04:29:06.615791668 (375h0m42.655364808s) [seg_pos={0 0}, upload=329615]: Done begin upload segment. Sep 22 04:29:06.615819902 (375h0m42.655393032s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616019633 (375h0m42.655592763s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616230955 (375h0m42.655804085s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616691882 (375h0m42.656265003s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616840605 (375h0m42.656413735s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616945585 (375h0m42.656518715s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617188929 (375h0m42.656762049s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617355596 (375h0m42.656928726s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617450227 (375h0m42.657023357s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617842974 (375h0m42.657416104s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617946411 (375h0m42.657519541s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618175438 (375h0m42.657748568s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618493012 (375h0m42.658066142s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618513361 (375h0m42.658086481s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.619067175 (375h0m42.658640305s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.622292537 (375h0m42.661865667s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:08.380779543 (375h0m44.420352673s) [upload=329615]: Next returned nil segment Sep 22 04:29:08.380797839 (375h0m44.420370969s) [upload=329615]: Waiting for error group managing segments... Sep 22 04:29:08.674700405 (375h0m44.714273525s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.709729657 (375h0m44.749302798s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.721175112 (375h0m44.760748232s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.728378538 (375h0m44.767951669s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.729967575 (375h0m44.769540705s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.736253273 (375h0m44.775826403s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.747689690 (375h0m44.787262821s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.768535595 (375h0m44.808108715s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.771670174 (375h0m44.811243294s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.772745321 (375h0m44.812318451s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.780813184 (375h0m44.820386314s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.789877134 (375h0m44.829450264s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:35.394668981 (375h1m11.434242111s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394720769 (375h1m11.434293899s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394765194 (375h1m11.434338324s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: rpc: dial tcp 145.79.227.3:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394804739 (375h1m11.434377869s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394821110 (375h1m11.43439424s) [seg_pos={0 0}, upload=329615]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:29:35.636648055 (375h1m11.676221175s) [upload=329615]: Done uploading segments... Sep 22 04:29:35.636675467 (375h1m11.676248587s) [upload=329615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.420064998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 6.210988053s FAIL (2025-09-22 04:31:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.211022528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X.X..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X...] duration: 6.343395212s FAIL (2025-09-22 04:30:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.343421393s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 5.907239083s FAIL (2025-09-22 04:30:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.907271236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....X...X.X.X.........XX..XX..X.X....XX........X.....X...X.] duration: 5.943169247s FAIL (2025-09-22 04:30:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:29:06.120000576 (375h0m42.159573706s) [upload=329615]: Starting upload Sep 22 04:29:06.120076982 (375h0m42.159650102s) [upload=329615]: Done starting upload Sep 22 04:29:06.120198202 (375h0m42.159771333s) [upload=329615]: Uploading segments... Sep 22 04:29:06.120374228 (375h0m42.159947368s) [upload=329615]: Got next segment. Inline: false Sep 22 04:29:06.120423472 (375h0m42.159996602s) [upload=329615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:29:06.120429143 (375h0m42.160002263s) [upload=329615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:29:06.615739730 (375h0m42.65531285s) [seg_pos={0 0}, upload=329615]: Begin upload segment... Sep 22 04:29:06.615791668 (375h0m42.655364808s) [seg_pos={0 0}, upload=329615]: Done begin upload segment. Sep 22 04:29:06.615819902 (375h0m42.655393032s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616019633 (375h0m42.655592763s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616230955 (375h0m42.655804085s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616691882 (375h0m42.656265003s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616840605 (375h0m42.656413735s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616945585 (375h0m42.656518715s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617188929 (375h0m42.656762049s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617355596 (375h0m42.656928726s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617450227 (375h0m42.657023357s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617842974 (375h0m42.657416104s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617946411 (375h0m42.657519541s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618175438 (375h0m42.657748568s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618493012 (375h0m42.658066142s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618513361 (375h0m42.658086481s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.619067175 (375h0m42.658640305s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.622292537 (375h0m42.661865667s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:08.380779543 (375h0m44.420352673s) [upload=329615]: Next returned nil segment Sep 22 04:29:08.380797839 (375h0m44.420370969s) [upload=329615]: Waiting for error group managing segments... Sep 22 04:29:08.674700405 (375h0m44.714273525s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.709729657 (375h0m44.749302798s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.721175112 (375h0m44.760748232s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.728378538 (375h0m44.767951669s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.729967575 (375h0m44.769540705s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.736253273 (375h0m44.775826403s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.747689690 (375h0m44.787262821s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.768535595 (375h0m44.808108715s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.771670174 (375h0m44.811243294s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.772745321 (375h0m44.812318451s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.780813184 (375h0m44.820386314s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.789877134 (375h0m44.829450264s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:35.394668981 (375h1m11.434242111s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394720769 (375h1m11.434293899s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394765194 (375h1m11.434338324s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: rpc: dial tcp 145.79.227.3:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394804739 (375h1m11.434377869s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394821110 (375h1m11.43439424s) [seg_pos={0 0}, upload=329615]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:29:35.636648055 (375h1m11.676221175s) [upload=329615]: Done uploading segments... Sep 22 04:29:35.636675467 (375h1m11.676248587s) [upload=329615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.943202831s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 5.951542596s FAIL (2025-09-22 04:30:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments... Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment... Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment. Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece... Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments... Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil> Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments... Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.95157597s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.X.X..XX.......X.X.X....X......XXXXXXX.X......X.X.X..X..] duration: 6.044385871s FAIL (2025-09-22 04:29:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.044412571s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X....X...X.X.X.........XX..XX..X.X....XX........X.....X...X] duration: 30.24450566s FAIL (2025-09-22 04:29:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:29:06.120000576 (375h0m42.159573706s) [upload=329615]: Starting upload Sep 22 04:29:06.120076982 (375h0m42.159650102s) [upload=329615]: Done starting upload Sep 22 04:29:06.120198202 (375h0m42.159771333s) [upload=329615]: Uploading segments... Sep 22 04:29:06.120374228 (375h0m42.159947368s) [upload=329615]: Got next segment. Inline: false Sep 22 04:29:06.120423472 (375h0m42.159996602s) [upload=329615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:29:06.120429143 (375h0m42.160002263s) [upload=329615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:29:06.615739730 (375h0m42.65531285s) [seg_pos={0 0}, upload=329615]: Begin upload segment... Sep 22 04:29:06.615791668 (375h0m42.655364808s) [seg_pos={0 0}, upload=329615]: Done begin upload segment. Sep 22 04:29:06.615819902 (375h0m42.655393032s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616019633 (375h0m42.655592763s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616230955 (375h0m42.655804085s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616691882 (375h0m42.656265003s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616840605 (375h0m42.656413735s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.616945585 (375h0m42.656518715s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617188929 (375h0m42.656762049s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617355596 (375h0m42.656928726s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617450227 (375h0m42.657023357s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617842974 (375h0m42.657416104s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.617946411 (375h0m42.657519541s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618175438 (375h0m42.657748568s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618493012 (375h0m42.658066142s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.618513361 (375h0m42.658086481s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.619067175 (375h0m42.658640305s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:06.622292537 (375h0m42.661865667s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Uploading piece... Sep 22 04:29:08.380779543 (375h0m44.420352673s) [upload=329615]: Next returned nil segment Sep 22 04:29:08.380797839 (375h0m44.420370969s) [upload=329615]: Waiting for error group managing segments... Sep 22 04:29:08.674700405 (375h0m44.714273525s) [noise=true , address=45.91.116.14:4189 , piece_id=NDOLRUMZCDPCQVN4Y4RQGYE5O3EQXWAXCWB32GEPLEYYEKTMB4UA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.709729657 (375h0m44.749302798s) [noise=true , address=45.91.116.6:4975 , piece_id=FJ22AMRPCLADOT3OLGZCRP3VXLW5B7INCO7BW3TVE557OGSZKVEA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.721175112 (375h0m44.760748232s) [noise=true , address=45.91.116.4:4939 , piece_id=WDQSKE2M26KINKWJLAN33VLBEIE7CXWIWGZ65ZEKFICE7BC336TA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.728378538 (375h0m44.767951669s) [noise=true , address=45.91.116.8:4048 , piece_id=HMWOKZXP2KAQZSACLGDDWTVUZGB4YOBR7D5M5OVSTEN4ODRASISQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.729967575 (375h0m44.769540705s) [noise=true , address=45.91.116.10:4082 , piece_id=OHNU2O5ZPCMF3VKGSQKC4ZNRBEEYHVIOSUESQ7K3HEB6LGH5ZMMQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.736253273 (375h0m44.775826403s) [noise=true , address=45.91.116.8:4045 , piece_id=5IZ7WDSHZBUJ2KYRA6YD432H66MTRSRJEMQHZ5EEH2S35GBFM4ZA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.747689690 (375h0m44.787262821s) [noise=true , address=45.91.116.5:4298 , piece_id=B75K6MFD7PZNLUBLWSC4FN6VTDMZUIKQE7NI2KQWDMELGSHPKABA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.768535595 (375h0m44.808108715s) [noise=true , address=45.91.116.4:4940 , piece_id=T34VQ4LYZ3X3C4HWRIHWB3UHXEGAK5XPMORS5EMISG7TAR6VTBQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.771670174 (375h0m44.811243294s) [noise=true , address=45.91.116.7:4010 , piece_id=M2XXFFVNMWWHA5OECIYSADXDOIKRF4XRGMUNRDIXFCSNVM4EWONQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.772745321 (375h0m44.812318451s) [noise=true , address=77.90.31.12:4589 , piece_id=BRL4J3EGYXCAE4YXCGKZ63SH6XQRT5FPCEQLJDF54CIL4ZJXLBHA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.780813184 (375h0m44.820386314s) [noise=true , address=45.91.116.11:4117 , piece_id=4HH242HH3KEMWCSJCQ5DOXDW6RZBUUHUTC3EFUEZ224QN6TXAYHQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:08.789877134 (375h0m44.829450264s) [noise=true , address=77.90.31.17:4770 , piece_id=535C25YEMTI6M67SJJUTBXQF67KK7FVLUVYBAL7GPKKPMIK5PB7A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: <nil> Sep 22 04:29:35.394668981 (375h1m11.434242111s) [noise=true , address=145.79.227.18:4005 , piece_id=BNZJUNGKE5U27BNL5G6DQZOZ3Z4EKY7OF7T565ONJRJP7GQB5KQQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394720769 (375h1m11.434293899s) [noise=true , address=145.79.227.17:4970 , piece_id=RYU6YD3BL63KIT3W3SPXQKLYQ46SIBIENGK74AMOMNCOFDE7RWRA, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394765194 (375h1m11.434338324s) [noise=true , address=145.79.227.3:4150 , piece_id=IQYHOAMICANTJ4SLDS2KUCPDGJCKO2JEXHV6WGDKHZISRXM6OZBQ, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: rpc: dial tcp 145.79.227.3:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394804739 (375h1m11.434377869s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEQ4NTRBJID4ZGKPNIFL23LRK23O6NOJTMXOIQ4CO2BC5K4YZT5A, seg_pos={0 0}, upload=329615]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:29:35.394821110 (375h1m11.43439424s) [seg_pos={0 0}, upload=329615]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:29:35.636648055 (375h1m11.676221175s) [upload=329615]: Done uploading segments... Sep 22 04:29:35.636675467 (375h1m11.676248587s) [upload=329615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.244538853s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)