Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X...........XX.....] duration: 6.935727921s FAIL (2025-10-06 23:04:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:59:54.702014331 (321h28m40.164106814s) [upload=282933]: Starting upload Oct 6 22:59:54.702104343 (321h28m40.164196826s) [upload=282933]: Done starting upload Oct 6 22:59:54.702222057 (321h28m40.16431454s) [upload=282933]: Uploading segments... Oct 6 22:59:54.702333680 (321h28m40.164426163s) [upload=282933]: Got next segment. Inline: false Oct 6 22:59:54.702403693 (321h28m40.164496176s) [upload=282933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:54.702411007 (321h28m40.16450349s) [upload=282933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:54.950723835 (321h28m40.412816318s) [seg_pos={0 0}, upload=282933]: Begin upload segment... Oct 6 22:59:54.950832923 (321h28m40.412925396s) [seg_pos={0 0}, upload=282933]: Done begin upload segment. Oct 6 22:59:54.950894781 (321h28m40.412987254s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951428066 (321h28m40.413520549s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951701657 (321h28m40.41379414s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952298664 (321h28m40.414391157s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952690530 (321h28m40.414783013s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952759842 (321h28m40.414852325s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953268962 (321h28m40.415361445s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953386826 (321h28m40.415479309s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953713879 (321h28m40.415806362s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954136494 (321h28m40.416228977s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954216355 (321h28m40.416308828s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954682753 (321h28m40.416775236s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954800929 (321h28m40.416893412s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955387456 (321h28m40.417479929s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955987779 (321h28m40.418080262s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.956564075 (321h28m40.418656558s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:56.570603436 (321h28m42.032695919s) [upload=282933]: Next returned nil segment Oct 6 22:59:56.570626519 (321h28m42.032719002s) [upload=282933]: Waiting for error group managing segments... Oct 6 22:59:56.928754072 (321h28m42.390846555s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.934964016 (321h28m42.39705651s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.945640235 (321h28m42.407732718s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.947979349 (321h28m42.410071823s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949424241 (321h28m42.411516724s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949861804 (321h28m42.411954287s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949972695 (321h28m42.412065178s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.951742295 (321h28m42.413834778s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.953872662 (321h28m42.415965145s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.954076059 (321h28m42.416168542s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.970066137 (321h28m42.43215861s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:23.983933742 (321h29m9.446026235s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:24.005052134 (321h29m9.467144637s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005233569 (321h29m9.467326052s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:48964->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005394526 (321h29m9.467487009s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59606->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005545624 (321h29m9.467638137s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:34112->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005581662 (321h29m9.467674145s) [seg_pos={0 0}, upload=282933]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 23:00:24.254694693 (321h29m9.716787176s) [upload=282933]: Done uploading segments... Oct 6 23:00:24.254730291 (321h29m9.716822764s) [upload=282933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.935823914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.173920489s FAIL (2025-10-06 23:03:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.173941699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................XX........X.....] duration: 6.190993868s FAIL (2025-10-06 23:03:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.191018684s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.879925916s FAIL (2025-10-06 23:03:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.880035475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X...........XX....] duration: 6.474440172s FAIL (2025-10-06 23:03:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:59:54.702014331 (321h28m40.164106814s) [upload=282933]: Starting upload Oct 6 22:59:54.702104343 (321h28m40.164196826s) [upload=282933]: Done starting upload Oct 6 22:59:54.702222057 (321h28m40.16431454s) [upload=282933]: Uploading segments... Oct 6 22:59:54.702333680 (321h28m40.164426163s) [upload=282933]: Got next segment. Inline: false Oct 6 22:59:54.702403693 (321h28m40.164496176s) [upload=282933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:54.702411007 (321h28m40.16450349s) [upload=282933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:54.950723835 (321h28m40.412816318s) [seg_pos={0 0}, upload=282933]: Begin upload segment... Oct 6 22:59:54.950832923 (321h28m40.412925396s) [seg_pos={0 0}, upload=282933]: Done begin upload segment. Oct 6 22:59:54.950894781 (321h28m40.412987254s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951428066 (321h28m40.413520549s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951701657 (321h28m40.41379414s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952298664 (321h28m40.414391157s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952690530 (321h28m40.414783013s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952759842 (321h28m40.414852325s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953268962 (321h28m40.415361445s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953386826 (321h28m40.415479309s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953713879 (321h28m40.415806362s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954136494 (321h28m40.416228977s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954216355 (321h28m40.416308828s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954682753 (321h28m40.416775236s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954800929 (321h28m40.416893412s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955387456 (321h28m40.417479929s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955987779 (321h28m40.418080262s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.956564075 (321h28m40.418656558s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:56.570603436 (321h28m42.032695919s) [upload=282933]: Next returned nil segment Oct 6 22:59:56.570626519 (321h28m42.032719002s) [upload=282933]: Waiting for error group managing segments... Oct 6 22:59:56.928754072 (321h28m42.390846555s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.934964016 (321h28m42.39705651s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.945640235 (321h28m42.407732718s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.947979349 (321h28m42.410071823s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949424241 (321h28m42.411516724s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949861804 (321h28m42.411954287s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949972695 (321h28m42.412065178s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.951742295 (321h28m42.413834778s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.953872662 (321h28m42.415965145s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.954076059 (321h28m42.416168542s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.970066137 (321h28m42.43215861s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:23.983933742 (321h29m9.446026235s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:24.005052134 (321h29m9.467144637s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005233569 (321h29m9.467326052s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:48964->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005394526 (321h29m9.467487009s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59606->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005545624 (321h29m9.467638137s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:34112->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005581662 (321h29m9.467674145s) [seg_pos={0 0}, upload=282933]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 23:00:24.254694693 (321h29m9.716787176s) [upload=282933]: Done uploading segments... Oct 6 23:00:24.254730291 (321h29m9.716822764s) [upload=282933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.474471232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.135523227s FAIL (2025-10-06 23:02:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.135558674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................XX........X....] duration: 6.162021756s FAIL (2025-10-06 23:02:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.16205052s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X...........XX...] duration: 6.180300049s FAIL (2025-10-06 23:02:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:59:54.702014331 (321h28m40.164106814s) [upload=282933]: Starting upload Oct 6 22:59:54.702104343 (321h28m40.164196826s) [upload=282933]: Done starting upload Oct 6 22:59:54.702222057 (321h28m40.16431454s) [upload=282933]: Uploading segments... Oct 6 22:59:54.702333680 (321h28m40.164426163s) [upload=282933]: Got next segment. Inline: false Oct 6 22:59:54.702403693 (321h28m40.164496176s) [upload=282933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:54.702411007 (321h28m40.16450349s) [upload=282933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:54.950723835 (321h28m40.412816318s) [seg_pos={0 0}, upload=282933]: Begin upload segment... Oct 6 22:59:54.950832923 (321h28m40.412925396s) [seg_pos={0 0}, upload=282933]: Done begin upload segment. Oct 6 22:59:54.950894781 (321h28m40.412987254s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951428066 (321h28m40.413520549s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951701657 (321h28m40.41379414s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952298664 (321h28m40.414391157s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952690530 (321h28m40.414783013s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952759842 (321h28m40.414852325s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953268962 (321h28m40.415361445s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953386826 (321h28m40.415479309s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953713879 (321h28m40.415806362s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954136494 (321h28m40.416228977s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954216355 (321h28m40.416308828s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954682753 (321h28m40.416775236s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954800929 (321h28m40.416893412s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955387456 (321h28m40.417479929s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955987779 (321h28m40.418080262s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.956564075 (321h28m40.418656558s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:56.570603436 (321h28m42.032695919s) [upload=282933]: Next returned nil segment Oct 6 22:59:56.570626519 (321h28m42.032719002s) [upload=282933]: Waiting for error group managing segments... Oct 6 22:59:56.928754072 (321h28m42.390846555s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.934964016 (321h28m42.39705651s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.945640235 (321h28m42.407732718s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.947979349 (321h28m42.410071823s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949424241 (321h28m42.411516724s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949861804 (321h28m42.411954287s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949972695 (321h28m42.412065178s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.951742295 (321h28m42.413834778s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.953872662 (321h28m42.415965145s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.954076059 (321h28m42.416168542s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.970066137 (321h28m42.43215861s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:23.983933742 (321h29m9.446026235s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:24.005052134 (321h29m9.467144637s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005233569 (321h29m9.467326052s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:48964->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005394526 (321h29m9.467487009s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59606->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005545624 (321h29m9.467638137s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:34112->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005581662 (321h29m9.467674145s) [seg_pos={0 0}, upload=282933]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 23:00:24.254694693 (321h29m9.716787176s) [upload=282933]: Done uploading segments... Oct 6 23:00:24.254730291 (321h29m9.716822764s) [upload=282933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.180339526s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XX........X...] duration: 6.906352276s FAIL (2025-10-06 23:02:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.906376111s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.264634083s FAIL (2025-10-06 23:01:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.264682655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X...........XX..] duration: 6.402253563s FAIL (2025-10-06 23:01:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:59:54.702014331 (321h28m40.164106814s) [upload=282933]: Starting upload Oct 6 22:59:54.702104343 (321h28m40.164196826s) [upload=282933]: Done starting upload Oct 6 22:59:54.702222057 (321h28m40.16431454s) [upload=282933]: Uploading segments... Oct 6 22:59:54.702333680 (321h28m40.164426163s) [upload=282933]: Got next segment. Inline: false Oct 6 22:59:54.702403693 (321h28m40.164496176s) [upload=282933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:54.702411007 (321h28m40.16450349s) [upload=282933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:54.950723835 (321h28m40.412816318s) [seg_pos={0 0}, upload=282933]: Begin upload segment... Oct 6 22:59:54.950832923 (321h28m40.412925396s) [seg_pos={0 0}, upload=282933]: Done begin upload segment. Oct 6 22:59:54.950894781 (321h28m40.412987254s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951428066 (321h28m40.413520549s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951701657 (321h28m40.41379414s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952298664 (321h28m40.414391157s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952690530 (321h28m40.414783013s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952759842 (321h28m40.414852325s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953268962 (321h28m40.415361445s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953386826 (321h28m40.415479309s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953713879 (321h28m40.415806362s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954136494 (321h28m40.416228977s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954216355 (321h28m40.416308828s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954682753 (321h28m40.416775236s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954800929 (321h28m40.416893412s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955387456 (321h28m40.417479929s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955987779 (321h28m40.418080262s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.956564075 (321h28m40.418656558s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:56.570603436 (321h28m42.032695919s) [upload=282933]: Next returned nil segment Oct 6 22:59:56.570626519 (321h28m42.032719002s) [upload=282933]: Waiting for error group managing segments... Oct 6 22:59:56.928754072 (321h28m42.390846555s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.934964016 (321h28m42.39705651s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.945640235 (321h28m42.407732718s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.947979349 (321h28m42.410071823s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949424241 (321h28m42.411516724s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949861804 (321h28m42.411954287s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949972695 (321h28m42.412065178s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.951742295 (321h28m42.413834778s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.953872662 (321h28m42.415965145s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.954076059 (321h28m42.416168542s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.970066137 (321h28m42.43215861s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:23.983933742 (321h29m9.446026235s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:24.005052134 (321h29m9.467144637s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005233569 (321h29m9.467326052s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:48964->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005394526 (321h29m9.467487009s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59606->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005545624 (321h29m9.467638137s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:34112->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005581662 (321h29m9.467674145s) [seg_pos={0 0}, upload=282933]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 23:00:24.254694693 (321h29m9.716787176s) [upload=282933]: Done uploading segments... Oct 6 23:00:24.254730291 (321h29m9.716822764s) [upload=282933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.40228866s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XX........X..] duration: 6.478777496s FAIL (2025-10-06 23:01:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.478807112s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X...........XX.] duration: 6.738691063s FAIL (2025-10-06 23:01:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:59:54.702014331 (321h28m40.164106814s) [upload=282933]: Starting upload Oct 6 22:59:54.702104343 (321h28m40.164196826s) [upload=282933]: Done starting upload Oct 6 22:59:54.702222057 (321h28m40.16431454s) [upload=282933]: Uploading segments... Oct 6 22:59:54.702333680 (321h28m40.164426163s) [upload=282933]: Got next segment. Inline: false Oct 6 22:59:54.702403693 (321h28m40.164496176s) [upload=282933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:54.702411007 (321h28m40.16450349s) [upload=282933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:54.950723835 (321h28m40.412816318s) [seg_pos={0 0}, upload=282933]: Begin upload segment... Oct 6 22:59:54.950832923 (321h28m40.412925396s) [seg_pos={0 0}, upload=282933]: Done begin upload segment. Oct 6 22:59:54.950894781 (321h28m40.412987254s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951428066 (321h28m40.413520549s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951701657 (321h28m40.41379414s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952298664 (321h28m40.414391157s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952690530 (321h28m40.414783013s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952759842 (321h28m40.414852325s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953268962 (321h28m40.415361445s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953386826 (321h28m40.415479309s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953713879 (321h28m40.415806362s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954136494 (321h28m40.416228977s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954216355 (321h28m40.416308828s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954682753 (321h28m40.416775236s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954800929 (321h28m40.416893412s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955387456 (321h28m40.417479929s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955987779 (321h28m40.418080262s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.956564075 (321h28m40.418656558s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:56.570603436 (321h28m42.032695919s) [upload=282933]: Next returned nil segment Oct 6 22:59:56.570626519 (321h28m42.032719002s) [upload=282933]: Waiting for error group managing segments... Oct 6 22:59:56.928754072 (321h28m42.390846555s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.934964016 (321h28m42.39705651s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.945640235 (321h28m42.407732718s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.947979349 (321h28m42.410071823s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949424241 (321h28m42.411516724s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949861804 (321h28m42.411954287s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949972695 (321h28m42.412065178s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.951742295 (321h28m42.413834778s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.953872662 (321h28m42.415965145s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.954076059 (321h28m42.416168542s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.970066137 (321h28m42.43215861s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:23.983933742 (321h29m9.446026235s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:24.005052134 (321h29m9.467144637s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005233569 (321h29m9.467326052s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:48964->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005394526 (321h29m9.467487009s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59606->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005545624 (321h29m9.467638137s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:34112->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005581662 (321h29m9.467674145s) [seg_pos={0 0}, upload=282933]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 23:00:24.254694693 (321h29m9.716787176s) [upload=282933]: Done uploading segments... Oct 6 23:00:24.254730291 (321h29m9.716822764s) [upload=282933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.738726792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.453104859s FAIL (2025-10-06 23:00:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.45316818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 22.195005351s FAIL (2025-10-06 23:00:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.195034936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X...........XX] duration: 30.251604863s FAIL (2025-10-06 23:00:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:59:54.702014331 (321h28m40.164106814s) [upload=282933]: Starting upload Oct 6 22:59:54.702104343 (321h28m40.164196826s) [upload=282933]: Done starting upload Oct 6 22:59:54.702222057 (321h28m40.16431454s) [upload=282933]: Uploading segments... Oct 6 22:59:54.702333680 (321h28m40.164426163s) [upload=282933]: Got next segment. Inline: false Oct 6 22:59:54.702403693 (321h28m40.164496176s) [upload=282933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:54.702411007 (321h28m40.16450349s) [upload=282933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:54.950723835 (321h28m40.412816318s) [seg_pos={0 0}, upload=282933]: Begin upload segment... Oct 6 22:59:54.950832923 (321h28m40.412925396s) [seg_pos={0 0}, upload=282933]: Done begin upload segment. Oct 6 22:59:54.950894781 (321h28m40.412987254s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951428066 (321h28m40.413520549s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.951701657 (321h28m40.41379414s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952298664 (321h28m40.414391157s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952690530 (321h28m40.414783013s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.952759842 (321h28m40.414852325s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953268962 (321h28m40.415361445s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953386826 (321h28m40.415479309s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.953713879 (321h28m40.415806362s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954136494 (321h28m40.416228977s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954216355 (321h28m40.416308828s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954682753 (321h28m40.416775236s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.954800929 (321h28m40.416893412s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955387456 (321h28m40.417479929s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.955987779 (321h28m40.418080262s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:54.956564075 (321h28m40.418656558s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Uploading piece... Oct 6 22:59:56.570603436 (321h28m42.032695919s) [upload=282933]: Next returned nil segment Oct 6 22:59:56.570626519 (321h28m42.032719002s) [upload=282933]: Waiting for error group managing segments... Oct 6 22:59:56.928754072 (321h28m42.390846555s) [noise=true , address=145.79.227.2:4113 , piece_id=7KA2JOZN4ZVHRZGID456MSBZ6OL2PG7VHPR6VH6XBL7NF6LWSUIQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.934964016 (321h28m42.39705651s) [noise=true , address=145.79.227.13:4801 , piece_id=Q457R4ZGNVC74BCNVYOQEMYXIMHNUJT4245C4RRVJQB6D4R5XVAA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.945640235 (321h28m42.407732718s) [noise=true , address=145.79.227.7:4295 , piece_id=NNGFA7XGYZHBIC5FGHIWFFVJZLRA3UMCF6CYWLI7CUXN5BPPGN5Q, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.947979349 (321h28m42.410071823s) [noise=true , address=145.79.227.14:4838 , piece_id=CX5SWLWMHDLZGWTM4ITNFJMPHIR6WOMFNFTMYDNCNZMFEJMNTSKQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949424241 (321h28m42.411516724s) [noise=true , address=145.79.227.5:4223 , piece_id=AH7POYCTIBGQ45NC6OII3UA4UQCWE7SN5I2THJPJOBJ5IGWNM75A, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949861804 (321h28m42.411954287s) [noise=true , address=145.79.227.16:4909 , piece_id=MLXH4YFZUD4HBO3TQKXOAO2TTCCDDZOGCORNZZU2E7TDID4TUEOA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.949972695 (321h28m42.412065178s) [noise=true , address=145.79.227.12:4767 , piece_id=3UWQ5YDIBP64HFNUFTKI3B3STSVSTA4MUZBYYZCD5ZOJWB6FIYPQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.951742295 (321h28m42.413834778s) [noise=true , address=145.79.227.11:4732 , piece_id=R7ATLU3V5EDWIB35NRPEVJWT2M3LXU57R6D34VR2LN22MYRX66JA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.953872662 (321h28m42.415965145s) [noise=true , address=145.79.227.17:4970 , piece_id=FRSO4D3RUI3ACRQG3Z3JL57CBZWCKQCT6N5LBZVXVNURCIIVSJSA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.954076059 (321h28m42.416168542s) [noise=true , address=145.79.227.7:4294 , piece_id=V2QQDDUIXMATDBRK7MFXATT3EV4ZQI6WNMWXTH2QLQVH3YZGD3QA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 22:59:56.970066137 (321h28m42.43215861s) [noise=true , address=145.79.227.4:4185 , piece_id=IRHKULK4BNVOXTNV3BXFC77P6XASX6MXW5M4LC4WZLQQUIQA66FA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:23.983933742 (321h29m9.446026235s) [noise=true , address=77.90.31.11:4555 , piece_id=FGKFTZWMVOAAKS27A3VMGICII2N4UYWU6OBQ2VYLYJS2TBGSGYDQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: <nil> Oct 6 23:00:24.005052134 (321h29m9.467144637s) [noise=true , address=45.91.116.14:4191 , piece_id=NJ3OL43H5UUCYM2YZLJFLFOUAOOYULSTXP5ADARUI2EDFIL2NSEA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005233569 (321h29m9.467326052s) [noise=true , address=77.90.31.14:4663 , piece_id=UABMUB3XMY4SLWS5J2OLDP6RHW457ZVSKARWWX55VJYJF6LCEKVQ, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:48964->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005394526 (321h29m9.467487009s) [noise=true , address=45.91.116.7:4010 , piece_id=KYJG4ENDVEMKWMSBORW233HVXWKRNGY2ZTE3GOHTR6YZ2Z6BHPIA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:59606->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005545624 (321h29m9.467638137s) [noise=true , address=77.90.31.16:4735 , piece_id=QWSY7MFWPZ6CESKRMBLTGZK65XBTUUKS55G3GAZZQ6CWMB3Z5DPA, seg_pos={0 0}, upload=282933]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:34112->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:00:24.005581662 (321h29m9.467674145s) [seg_pos={0 0}, upload=282933]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 23:00:24.254694693 (321h29m9.716787176s) [upload=282933]: Done uploading segments... Oct 6 23:00:24.254730291 (321h29m9.716822764s) [upload=282933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.251622667s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................XX........X.] duration: 6.210088711s FAIL (2025-10-06 23:00:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.210120773s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...........X] duration: 30.241939742s FAIL (2025-10-06 22:59:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== t 6 22:59:24.239300943 (321h28m9.701393426s) [upload=282929]: Starting upload Oct 6 22:59:24.239394501 (321h28m9.701486984s) [upload=282929]: Done starting upload Oct 6 22:59:24.239546681 (321h28m9.701639174s) [upload=282929]: Uploading segments... Oct 6 22:59:24.239732505 (321h28m9.701824998s) [upload=282929]: Got next segment. Inline: false Oct 6 22:59:24.239790566 (321h28m9.701883039s) [upload=282929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:59:24.239797629 (321h28m9.701890112s) [upload=282929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:59:24.490684828 (321h28m9.952777321s) [seg_pos={0 0}, upload=282929]: Begin upload segment... Oct 6 22:59:24.490752326 (321h28m9.952844809s) [seg_pos={0 0}, upload=282929]: Done begin upload segment. Oct 6 22:59:24.490776733 (321h28m9.952869216s) [noise=true , address=145.79.227.11:4731 , piece_id=SCWQAZ5M5Y4O2RGRZIINYFUKFSK44HHGLT75GN5NOPL7LQBTFBOA, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.490788446 (321h28m9.952880919s) [noise=true , address=145.79.227.7:4294 , piece_id=532NK6AIYUROLGFPTKIFUIZY4OBCYF2I4NPY3BVQOYTHXIUH6HEQ, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.490855352 (321h28m9.952947835s) [noise=true , address=145.79.227.13:4801 , piece_id=2XSWWAQIIKC77Z5DH2RMIVJGPKBAN2PQSFKIPXJGDOLGWATFMHEQ, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.491370323 (321h28m9.953462806s) [noise=true , address=145.79.227.6:4257 , piece_id=LLXKSWF5QYHZZMGZXJKFX6ZJTZOLDI3TBLGRXRI7DK7JIDPKCVWA, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.492189102 (321h28m9.954281586s) [noise=true , address=45.91.116.12:4227 , piece_id=NAUUV6XEUJXLEFZ43EPJZPELKU2GYW3VTWPGBSWIVUKMQL6KGFIQ, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.492587992 (321h28m9.954680465s) [noise=true , address=77.90.31.10:4518 , piece_id=3CG7ZJTR4U7WPNMMAV5UO7MKSGD775AQZ2E6NZHHZXB3FTDO2R4A, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.493289277 (321h28m9.95538175s) [noise=true , address=145.79.227.2:4115 , piece_id=PHBWN3RQ37BXBQKGY3GKT2R5EVYDK4HUOJ4R7YMP5RHICWMTXTVQ, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.493305228 (321h28m9.955397711s) [noise=true , address=45.91.116.12:4225 , piece_id=EHZFLKDQI54QWYVGTZH2Y3AAVYKSBHGBS3PMV7OH6CM4MOEUPH3Q, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.493748051 (321h28m9.955840524s) [noise=true , address=77.90.31.14:4663 , piece_id=MIQBHZ7VHCJVY3RHZEWALPSC5NJBFH5ZOWF2ZMNJLOUEQ7LBBGXA, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.493773559 (321h28m9.955866032s) [noise=true , address=145.79.227.6:4259 , piece_id=TV7HTDGZM2XEDWKKB4FEZU252YRNWWVLHLCNNAICXZ7MYNOPOMVA, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.494286987 (321h28m9.95637947s) [noise=true , address=77.90.31.17:4772 , piece_id=QWYE4RWLYSCGJT46DKURJEAWQNYTBK5QJ7QJH7UXWJX4RNLHHMFA, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.494742975 (321h28m9.956835458s) [noise=true , address=77.90.31.13:4628 , piece_id=U6T5V5NCTOT2JJIYGX6DC3TUVTJMF7FC377ETUOYQNKF3QQOCD4Q, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.494989134 (321h28m9.957081617s) [noise=true , address=45.91.116.5:4299 , piece_id=VMC32Z5HEIW35PTIJHRUZTNMTONGTHBBFFYSTD7NEYLCWNLMBSWA, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.495214293 (321h28m9.957306786s) [noise=true , address=145.79.227.5:4223 , piece_id=L2QJVUJT22FHUZAAPQGAP4GRQOXCJECP6CSZX7IQW3PQGQ25B7TQ, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.495442748 (321h28m9.957535241s) [noise=true , address=145.79.227.15:4876 , piece_id=KDTLDBLGL6O7FLMRGVU2WQPFT5ZWEDRT2YBSCPYFKFT3LOL23GRQ, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:24.496378720 (321h28m9.958471203s) [noise=true , address=145.79.227.17:4971 , piece_id=63KBOVT2MS7TL5GOBQJ4ZBSGKBEDAR4AHQ4HDPYSV6DJY6WA276A, seg_pos={0 0}, upload=282929]: Uploading piece... Oct 6 22:59:26.078301493 (321h28m11.540393977s) [upload=282929]: Next returned nil segment Oct 6 22:59:26.078320580 (321h28m11.540413063s) [upload=282929]: Waiting for error group managing segments... Oct 6 22:59:26.456657985 (321h28m11.918750468s) [noise=true , address=145.79.227.5:4223 , piece_id=L2QJVUJT22FHUZAAPQGAP4GRQOXCJECP6CSZX7IQW3PQGQ25B7TQ, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:26.460046406 (321h28m11.922138889s) [noise=true , address=145.79.227.13:4801 , piece_id=2XSWWAQIIKC77Z5DH2RMIVJGPKBAN2PQSFKIPXJGDOLGWATFMHEQ, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:26.463201815 (321h28m11.925294298s) [noise=true , address=145.79.227.11:4731 , piece_id=SCWQAZ5M5Y4O2RGRZIINYFUKFSK44HHGLT75GN5NOPL7LQBTFBOA, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:26.469790541 (321h28m11.931883014s) [noise=true , address=145.79.227.6:4259 , piece_id=TV7HTDGZM2XEDWKKB4FEZU252YRNWWVLHLCNNAICXZ7MYNOPOMVA, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:26.470839468 (321h28m11.932931941s) [noise=true , address=145.79.227.15:4876 , piece_id=KDTLDBLGL6O7FLMRGVU2WQPFT5ZWEDRT2YBSCPYFKFT3LOL23GRQ, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:26.489274662 (321h28m11.951367145s) [noise=true , address=145.79.227.6:4257 , piece_id=LLXKSWF5QYHZZMGZXJKFX6ZJTZOLDI3TBLGRXRI7DK7JIDPKCVWA, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:26.490987684 (321h28m11.953080157s) [noise=true , address=145.79.227.7:4294 , piece_id=532NK6AIYUROLGFPTKIFUIZY4OBCYF2I4NPY3BVQOYTHXIUH6HEQ, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:26.494621984 (321h28m11.956714467s) [noise=true , address=145.79.227.17:4971 , piece_id=63KBOVT2MS7TL5GOBQJ4ZBSGKBEDAR4AHQ4HDPYSV6DJY6WA276A, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:26.498609997 (321h28m11.96070247s) [noise=true , address=145.79.227.2:4115 , piece_id=PHBWN3RQ37BXBQKGY3GKT2R5EVYDK4HUOJ4R7YMP5RHICWMTXTVQ, seg_pos={0 0}, upload=282929]: Done uploading piece. err: <nil> Oct 6 22:59:53.545553555 (321h28m39.007646039s) [noise=true , address=77.90.31.14:4663 , piece_id=MIQBHZ7VHCJVY3RHZEWALPSC5NJBFH5ZOWF2ZMNJLOUEQ7LBBGXA, seg_pos={0 0}, upload=282929]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:59:53.545719081 (321h28m39.007811564s) [noise=true , address=77.90.31.17:4772 , piece_id=QWYE4RWLYSCGJT46DKURJEAWQNYTBK5QJ7QJH7UXWJX4RNLHHMFA, seg_pos={0 0}, upload=282929]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 188.166.225.170:42044->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:59:53.545757734 (321h28m39.007850218s) [noise=true , address=45.91.116.12:4225 , piece_id=EHZFLKDQI54QWYVGTZH2Y3AAVYKSBHGBS3PMV7OH6CM4MOEUPH3Q, seg_pos={0 0}, upload=282929]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:42050->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:59:53.545771521 (321h28m39.007863994s) [noise=true , address=77.90.31.10:4518 , piece_id=3CG7ZJTR4U7WPNMMAV5UO7MKSGD775AQZ2E6NZHHZXB3FTDO2R4A, seg_pos={0 0}, upload=282929]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 188.166.225.170:38000->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:59:53.545926807 (321h28m39.00801928s) [noise=true , address=77.90.31.13:4628 , piece_id=U6T5V5NCTOT2JJIYGX6DC3TUVTJMF7FC377ETUOYQNKF3QQOCD4Q, seg_pos={0 0}, upload=282929]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:45496->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:59:53.546058638 (321h28m39.008151121s) [noise=true , address=45.91.116.5:4299 , piece_id=VMC32Z5HEIW35PTIJHRUZTNMTONGTHBBFFYSTD7NEYLCWNLMBSWA, seg_pos={0 0}, upload=282929]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:36086->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:59:53.546134111 (321h28m39.008226594s) [noise=true , address=45.91.116.12:4227 , piece_id=NAUUV6XEUJXLEFZ43EPJZPELKU2GYW3VTWPGBSWIVUKMQL6KGFIQ, seg_pos={0 0}, upload=282929]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:48682->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:59:53.546176762 (321h28m39.008269245s) [seg_pos={0 0}, upload=282929]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 22:59:53.785152762 (321h28m39.247245235s) [upload=282929]: Done uploading segments... Oct 6 22:59:53.785188841 (321h28m39.247281324s) [upload=282929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.241966982s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XX........X] duration: 30.00059916s FAIL (2025-10-06 22:59:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000622764s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.009450452s FAIL (2025-10-06 22:59:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.009483937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.173596112s FAIL (2025-10-06 22:58:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.173639694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.947901381s FAIL (2025-10-06 22:58:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.947939144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.184790485s FAIL (2025-10-06 22:58:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.184885015s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 6.256462404s FAIL (2025-10-06 22:58:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.256503571s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 6.214060233s FAIL (2025-10-06 22:57:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.214086493s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.735595109s FAIL (2025-10-06 22:57:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.735622651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.085195021s FAIL (2025-10-06 22:57:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.085221882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.434624701s FAIL (2025-10-06 22:57:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.434648057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.216293725s FAIL (2025-10-06 22:57:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.21632787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 6.231609266s FAIL (2025-10-06 22:56:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.231665092s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.270769304s FAIL (2025-10-06 22:56:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.270802456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.118723841s FAIL (2025-10-06 22:56:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.118757035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.178240752s FAIL (2025-10-06 22:56:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.178276622s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 6.379296646s FAIL (2025-10-06 22:56:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.379323567s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.061849124s FAIL (2025-10-06 22:55:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.061912315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 6.415589645s FAIL (2025-10-06 22:55:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.415620615s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.472616773s FAIL (2025-10-06 22:55:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.472648886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.627495027s FAIL (2025-10-06 22:54:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.62751732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.221441869s FAIL (2025-10-06 22:54:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.221467517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.334995336s FAIL (2025-10-06 22:54:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.335025794s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.098713171s FAIL (2025-10-06 22:54:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.09874417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 6.156688616s FAIL (2025-10-06 22:54:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.156713332s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.199404522s FAIL (2025-10-06 22:53:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.199426864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.91619111s FAIL (2025-10-06 22:53:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.916257137s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 6.419174512s FAIL (2025-10-06 22:53:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.419243604s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.501139538s FAIL (2025-10-06 22:53:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.501169734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.308395922s FAIL (2025-10-06 22:53:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.308419237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 6.223396199s FAIL (2025-10-06 22:53:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.223425645s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.252529626s FAIL (2025-10-06 22:53:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.252557871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.667218052s FAIL (2025-10-06 22:53:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 14.66724352s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)