Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........XX............X....X........X...X........XX......] duration: 5.909850962s trace: https://jaeger.dev.storj.io/trace/3933635ea304ab04 FAIL (2025-11-13 23:29:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:23:32.222110443 (111h19m30.773580904s) [upload=97944]: Starting upload Nov 13 23:23:32.222193351 (111h19m30.773663812s) [upload=97944]: Done starting upload Nov 13 23:23:32.222434801 (111h19m30.773905252s) [upload=97944]: Uploading segments... Nov 13 23:23:32.222535482 (111h19m30.774005943s) [upload=97944]: Got next segment. Inline: false Nov 13 23:23:32.222570429 (111h19m30.77404089s) [upload=97944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:23:32.222575719 (111h19m30.77404617s) [upload=97944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:23:32.489214273 (111h19m31.040684734s) [seg_pos={0 0}, upload=97944]: Begin upload segment... Nov 13 23:23:32.489262244 (111h19m31.040732695s) [seg_pos={0 0}, upload=97944]: Done begin upload segment. Nov 13 23:23:32.489288524 (111h19m31.040758975s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.489365290 (111h19m31.040835751s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490176184 (111h19m31.041646635s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490619007 (111h19m31.042089459s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490959626 (111h19m31.042430077s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491307619 (111h19m31.04277808s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491679226 (111h19m31.043149677s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.492333031 (111h19m31.043803492s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493445790 (111h19m31.044916251s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493939951 (111h19m31.045410412s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495108186 (111h19m31.046578647s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495243934 (111h19m31.046714395s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.496459299 (111h19m31.04792976s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.497730238 (111h19m31.049200689s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.498902531 (111h19m31.050372982s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.500159745 (111h19m31.051630206s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:34.168943842 (111h19m32.720414303s) [upload=97944]: Next returned nil segment Nov 13 23:23:34.168968589 (111h19m32.72043905s) [upload=97944]: Waiting for error group managing segments... Nov 13 23:23:34.566926491 (111h19m33.118396952s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.567727145 (111h19m33.119197596s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.584836114 (111h19m33.136306575s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.587038137 (111h19m33.138508598s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.603143174 (111h19m33.154613625s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.633392504 (111h19m33.184862955s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.646566731 (111h19m33.198037182s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.655804502 (111h19m33.207274963s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.678400780 (111h19m33.229871231s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:55.588855093 (111h19m54.140325554s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.058280180 (111h19m59.609750631s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.487675183 (111h20m0.039145654s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.488290434 (111h20m0.039760905s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection Nov 13 23:24:01.488833839 (111h20m0.0403043s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection Nov 13 23:24:01.489149691 (111h20m0.040620152s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection Nov 13 23:24:01.489463719 (111h20m0.04093418s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:01.489493796 (111h20m0.040964247s) [seg_pos={0 0}, upload=97944]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:24:01.748156531 (111h20m0.299627002s) [upload=97944]: Done uploading segments... Nov 13 23:24:01.748206497 (111h20m0.299676958s) [upload=97944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.909879768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.809886154s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:28:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.809906113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.689608172s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:28:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.689633431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............XX............X....X........X...X........XX.....] duration: 5.856576723s trace: https://jaeger.dev.storj.io/trace/3933635ea304ab04 FAIL (2025-11-13 23:28:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:23:32.222110443 (111h19m30.773580904s) [upload=97944]: Starting upload Nov 13 23:23:32.222193351 (111h19m30.773663812s) [upload=97944]: Done starting upload Nov 13 23:23:32.222434801 (111h19m30.773905252s) [upload=97944]: Uploading segments... Nov 13 23:23:32.222535482 (111h19m30.774005943s) [upload=97944]: Got next segment. Inline: false Nov 13 23:23:32.222570429 (111h19m30.77404089s) [upload=97944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:23:32.222575719 (111h19m30.77404617s) [upload=97944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:23:32.489214273 (111h19m31.040684734s) [seg_pos={0 0}, upload=97944]: Begin upload segment... Nov 13 23:23:32.489262244 (111h19m31.040732695s) [seg_pos={0 0}, upload=97944]: Done begin upload segment. Nov 13 23:23:32.489288524 (111h19m31.040758975s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.489365290 (111h19m31.040835751s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490176184 (111h19m31.041646635s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490619007 (111h19m31.042089459s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490959626 (111h19m31.042430077s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491307619 (111h19m31.04277808s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491679226 (111h19m31.043149677s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.492333031 (111h19m31.043803492s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493445790 (111h19m31.044916251s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493939951 (111h19m31.045410412s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495108186 (111h19m31.046578647s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495243934 (111h19m31.046714395s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.496459299 (111h19m31.04792976s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.497730238 (111h19m31.049200689s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.498902531 (111h19m31.050372982s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.500159745 (111h19m31.051630206s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:34.168943842 (111h19m32.720414303s) [upload=97944]: Next returned nil segment Nov 13 23:23:34.168968589 (111h19m32.72043905s) [upload=97944]: Waiting for error group managing segments... Nov 13 23:23:34.566926491 (111h19m33.118396952s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.567727145 (111h19m33.119197596s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.584836114 (111h19m33.136306575s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.587038137 (111h19m33.138508598s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.603143174 (111h19m33.154613625s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.633392504 (111h19m33.184862955s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.646566731 (111h19m33.198037182s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.655804502 (111h19m33.207274963s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.678400780 (111h19m33.229871231s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:55.588855093 (111h19m54.140325554s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.058280180 (111h19m59.609750631s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.487675183 (111h20m0.039145654s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.488290434 (111h20m0.039760905s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection Nov 13 23:24:01.488833839 (111h20m0.0403043s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection Nov 13 23:24:01.489149691 (111h20m0.040620152s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection Nov 13 23:24:01.489463719 (111h20m0.04093418s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:01.489493796 (111h20m0.040964247s) [seg_pos={0 0}, upload=97944]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:24:01.748156531 (111h20m0.299627002s) [upload=97944]: Done uploading segments... Nov 13 23:24:01.748206497 (111h20m0.299676958s) [upload=97944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.856609104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.741631408s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:27:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.741670324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............XX............X....X........X...X........XX....] duration: 5.921969496s trace: https://jaeger.dev.storj.io/trace/3933635ea304ab04 FAIL (2025-11-13 23:26:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:23:32.222110443 (111h19m30.773580904s) [upload=97944]: Starting upload Nov 13 23:23:32.222193351 (111h19m30.773663812s) [upload=97944]: Done starting upload Nov 13 23:23:32.222434801 (111h19m30.773905252s) [upload=97944]: Uploading segments... Nov 13 23:23:32.222535482 (111h19m30.774005943s) [upload=97944]: Got next segment. Inline: false Nov 13 23:23:32.222570429 (111h19m30.77404089s) [upload=97944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:23:32.222575719 (111h19m30.77404617s) [upload=97944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:23:32.489214273 (111h19m31.040684734s) [seg_pos={0 0}, upload=97944]: Begin upload segment... Nov 13 23:23:32.489262244 (111h19m31.040732695s) [seg_pos={0 0}, upload=97944]: Done begin upload segment. Nov 13 23:23:32.489288524 (111h19m31.040758975s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.489365290 (111h19m31.040835751s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490176184 (111h19m31.041646635s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490619007 (111h19m31.042089459s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490959626 (111h19m31.042430077s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491307619 (111h19m31.04277808s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491679226 (111h19m31.043149677s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.492333031 (111h19m31.043803492s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493445790 (111h19m31.044916251s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493939951 (111h19m31.045410412s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495108186 (111h19m31.046578647s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495243934 (111h19m31.046714395s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.496459299 (111h19m31.04792976s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.497730238 (111h19m31.049200689s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.498902531 (111h19m31.050372982s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.500159745 (111h19m31.051630206s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:34.168943842 (111h19m32.720414303s) [upload=97944]: Next returned nil segment Nov 13 23:23:34.168968589 (111h19m32.72043905s) [upload=97944]: Waiting for error group managing segments... Nov 13 23:23:34.566926491 (111h19m33.118396952s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.567727145 (111h19m33.119197596s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.584836114 (111h19m33.136306575s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.587038137 (111h19m33.138508598s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.603143174 (111h19m33.154613625s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.633392504 (111h19m33.184862955s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.646566731 (111h19m33.198037182s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.655804502 (111h19m33.207274963s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.678400780 (111h19m33.229871231s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:55.588855093 (111h19m54.140325554s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.058280180 (111h19m59.609750631s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.487675183 (111h20m0.039145654s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.488290434 (111h20m0.039760905s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection Nov 13 23:24:01.488833839 (111h20m0.0403043s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection Nov 13 23:24:01.489149691 (111h20m0.040620152s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection Nov 13 23:24:01.489463719 (111h20m0.04093418s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:01.489493796 (111h20m0.040964247s) [seg_pos={0 0}, upload=97944]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:24:01.748156531 (111h20m0.299627002s) [upload=97944]: Done uploading segments... Nov 13 23:24:01.748206497 (111h20m0.299676958s) [upload=97944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.921994132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............XX............X....X........X...X........XX...] duration: 5.97693002s trace: https://jaeger.dev.storj.io/trace/3933635ea304ab04 FAIL (2025-11-13 23:26:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:23:32.222110443 (111h19m30.773580904s) [upload=97944]: Starting upload Nov 13 23:23:32.222193351 (111h19m30.773663812s) [upload=97944]: Done starting upload Nov 13 23:23:32.222434801 (111h19m30.773905252s) [upload=97944]: Uploading segments... Nov 13 23:23:32.222535482 (111h19m30.774005943s) [upload=97944]: Got next segment. Inline: false Nov 13 23:23:32.222570429 (111h19m30.77404089s) [upload=97944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:23:32.222575719 (111h19m30.77404617s) [upload=97944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:23:32.489214273 (111h19m31.040684734s) [seg_pos={0 0}, upload=97944]: Begin upload segment... Nov 13 23:23:32.489262244 (111h19m31.040732695s) [seg_pos={0 0}, upload=97944]: Done begin upload segment. Nov 13 23:23:32.489288524 (111h19m31.040758975s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.489365290 (111h19m31.040835751s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490176184 (111h19m31.041646635s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490619007 (111h19m31.042089459s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490959626 (111h19m31.042430077s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491307619 (111h19m31.04277808s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491679226 (111h19m31.043149677s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.492333031 (111h19m31.043803492s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493445790 (111h19m31.044916251s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493939951 (111h19m31.045410412s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495108186 (111h19m31.046578647s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495243934 (111h19m31.046714395s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.496459299 (111h19m31.04792976s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.497730238 (111h19m31.049200689s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.498902531 (111h19m31.050372982s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.500159745 (111h19m31.051630206s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:34.168943842 (111h19m32.720414303s) [upload=97944]: Next returned nil segment Nov 13 23:23:34.168968589 (111h19m32.72043905s) [upload=97944]: Waiting for error group managing segments... Nov 13 23:23:34.566926491 (111h19m33.118396952s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.567727145 (111h19m33.119197596s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.584836114 (111h19m33.136306575s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.587038137 (111h19m33.138508598s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.603143174 (111h19m33.154613625s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.633392504 (111h19m33.184862955s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.646566731 (111h19m33.198037182s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.655804502 (111h19m33.207274963s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.678400780 (111h19m33.229871231s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:55.588855093 (111h19m54.140325554s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.058280180 (111h19m59.609750631s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.487675183 (111h20m0.039145654s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.488290434 (111h20m0.039760905s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection Nov 13 23:24:01.488833839 (111h20m0.0403043s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection Nov 13 23:24:01.489149691 (111h20m0.040620152s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection Nov 13 23:24:01.489463719 (111h20m0.04093418s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:01.489493796 (111h20m0.040964247s) [seg_pos={0 0}, upload=97944]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:24:01.748156531 (111h20m0.299627002s) [upload=97944]: Done uploading segments... Nov 13 23:24:01.748206497 (111h20m0.299676958s) [upload=97944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.976958204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.784552656s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:26:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.784573757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............XX............X....X........X...X........XX..] duration: 6.059636401s trace: https://jaeger.dev.storj.io/trace/3933635ea304ab04 FAIL (2025-11-13 23:25:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:23:32.222110443 (111h19m30.773580904s) [upload=97944]: Starting upload Nov 13 23:23:32.222193351 (111h19m30.773663812s) [upload=97944]: Done starting upload Nov 13 23:23:32.222434801 (111h19m30.773905252s) [upload=97944]: Uploading segments... Nov 13 23:23:32.222535482 (111h19m30.774005943s) [upload=97944]: Got next segment. Inline: false Nov 13 23:23:32.222570429 (111h19m30.77404089s) [upload=97944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:23:32.222575719 (111h19m30.77404617s) [upload=97944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:23:32.489214273 (111h19m31.040684734s) [seg_pos={0 0}, upload=97944]: Begin upload segment... Nov 13 23:23:32.489262244 (111h19m31.040732695s) [seg_pos={0 0}, upload=97944]: Done begin upload segment. Nov 13 23:23:32.489288524 (111h19m31.040758975s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.489365290 (111h19m31.040835751s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490176184 (111h19m31.041646635s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490619007 (111h19m31.042089459s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490959626 (111h19m31.042430077s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491307619 (111h19m31.04277808s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491679226 (111h19m31.043149677s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.492333031 (111h19m31.043803492s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493445790 (111h19m31.044916251s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493939951 (111h19m31.045410412s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495108186 (111h19m31.046578647s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495243934 (111h19m31.046714395s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.496459299 (111h19m31.04792976s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.497730238 (111h19m31.049200689s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.498902531 (111h19m31.050372982s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.500159745 (111h19m31.051630206s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:34.168943842 (111h19m32.720414303s) [upload=97944]: Next returned nil segment Nov 13 23:23:34.168968589 (111h19m32.72043905s) [upload=97944]: Waiting for error group managing segments... Nov 13 23:23:34.566926491 (111h19m33.118396952s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.567727145 (111h19m33.119197596s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.584836114 (111h19m33.136306575s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.587038137 (111h19m33.138508598s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.603143174 (111h19m33.154613625s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.633392504 (111h19m33.184862955s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.646566731 (111h19m33.198037182s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.655804502 (111h19m33.207274963s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.678400780 (111h19m33.229871231s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:55.588855093 (111h19m54.140325554s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.058280180 (111h19m59.609750631s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.487675183 (111h20m0.039145654s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.488290434 (111h20m0.039760905s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection Nov 13 23:24:01.488833839 (111h20m0.0403043s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection Nov 13 23:24:01.489149691 (111h20m0.040620152s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection Nov 13 23:24:01.489463719 (111h20m0.04093418s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:01.489493796 (111h20m0.040964247s) [seg_pos={0 0}, upload=97944]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:24:01.748156531 (111h20m0.299627002s) [upload=97944]: Done uploading segments... Nov 13 23:24:01.748206497 (111h20m0.299676958s) [upload=97944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.059747723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.742198095s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:25:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.742238301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.788169932s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.788219767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................XX............X....X........X...X........XX.] duration: 5.946874669s trace: https://jaeger.dev.storj.io/trace/3933635ea304ab04 FAIL (2025-11-13 23:24:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:23:32.222110443 (111h19m30.773580904s) [upload=97944]: Starting upload Nov 13 23:23:32.222193351 (111h19m30.773663812s) [upload=97944]: Done starting upload Nov 13 23:23:32.222434801 (111h19m30.773905252s) [upload=97944]: Uploading segments... Nov 13 23:23:32.222535482 (111h19m30.774005943s) [upload=97944]: Got next segment. Inline: false Nov 13 23:23:32.222570429 (111h19m30.77404089s) [upload=97944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:23:32.222575719 (111h19m30.77404617s) [upload=97944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:23:32.489214273 (111h19m31.040684734s) [seg_pos={0 0}, upload=97944]: Begin upload segment... Nov 13 23:23:32.489262244 (111h19m31.040732695s) [seg_pos={0 0}, upload=97944]: Done begin upload segment. Nov 13 23:23:32.489288524 (111h19m31.040758975s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.489365290 (111h19m31.040835751s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490176184 (111h19m31.041646635s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490619007 (111h19m31.042089459s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490959626 (111h19m31.042430077s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491307619 (111h19m31.04277808s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491679226 (111h19m31.043149677s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.492333031 (111h19m31.043803492s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493445790 (111h19m31.044916251s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493939951 (111h19m31.045410412s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495108186 (111h19m31.046578647s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495243934 (111h19m31.046714395s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.496459299 (111h19m31.04792976s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.497730238 (111h19m31.049200689s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.498902531 (111h19m31.050372982s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.500159745 (111h19m31.051630206s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:34.168943842 (111h19m32.720414303s) [upload=97944]: Next returned nil segment Nov 13 23:23:34.168968589 (111h19m32.72043905s) [upload=97944]: Waiting for error group managing segments... Nov 13 23:23:34.566926491 (111h19m33.118396952s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.567727145 (111h19m33.119197596s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.584836114 (111h19m33.136306575s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.587038137 (111h19m33.138508598s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.603143174 (111h19m33.154613625s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.633392504 (111h19m33.184862955s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.646566731 (111h19m33.198037182s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.655804502 (111h19m33.207274963s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.678400780 (111h19m33.229871231s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:55.588855093 (111h19m54.140325554s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.058280180 (111h19m59.609750631s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.487675183 (111h20m0.039145654s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.488290434 (111h20m0.039760905s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection Nov 13 23:24:01.488833839 (111h20m0.0403043s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection Nov 13 23:24:01.489149691 (111h20m0.040620152s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection Nov 13 23:24:01.489463719 (111h20m0.04093418s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:01.489493796 (111h20m0.040964247s) [seg_pos={0 0}, upload=97944]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:24:01.748156531 (111h20m0.299627002s) [upload=97944]: Done uploading segments... Nov 13 23:24:01.748206497 (111h20m0.299676958s) [upload=97944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.946895508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.253680525s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:24:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.253715322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................XX............X....X........X...X........XX] duration: 30.261301579s trace: https://jaeger.dev.storj.io/trace/3933635ea304ab04 FAIL (2025-11-13 23:24:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:23:32.222110443 (111h19m30.773580904s) [upload=97944]: Starting upload Nov 13 23:23:32.222193351 (111h19m30.773663812s) [upload=97944]: Done starting upload Nov 13 23:23:32.222434801 (111h19m30.773905252s) [upload=97944]: Uploading segments... Nov 13 23:23:32.222535482 (111h19m30.774005943s) [upload=97944]: Got next segment. Inline: false Nov 13 23:23:32.222570429 (111h19m30.77404089s) [upload=97944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:23:32.222575719 (111h19m30.77404617s) [upload=97944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:23:32.489214273 (111h19m31.040684734s) [seg_pos={0 0}, upload=97944]: Begin upload segment... Nov 13 23:23:32.489262244 (111h19m31.040732695s) [seg_pos={0 0}, upload=97944]: Done begin upload segment. Nov 13 23:23:32.489288524 (111h19m31.040758975s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.489365290 (111h19m31.040835751s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490176184 (111h19m31.041646635s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490619007 (111h19m31.042089459s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490959626 (111h19m31.042430077s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491307619 (111h19m31.04277808s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491679226 (111h19m31.043149677s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.492333031 (111h19m31.043803492s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493445790 (111h19m31.044916251s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493939951 (111h19m31.045410412s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495108186 (111h19m31.046578647s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495243934 (111h19m31.046714395s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.496459299 (111h19m31.04792976s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.497730238 (111h19m31.049200689s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.498902531 (111h19m31.050372982s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.500159745 (111h19m31.051630206s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:34.168943842 (111h19m32.720414303s) [upload=97944]: Next returned nil segment Nov 13 23:23:34.168968589 (111h19m32.72043905s) [upload=97944]: Waiting for error group managing segments... Nov 13 23:23:34.566926491 (111h19m33.118396952s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.567727145 (111h19m33.119197596s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.584836114 (111h19m33.136306575s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.587038137 (111h19m33.138508598s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.603143174 (111h19m33.154613625s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.633392504 (111h19m33.184862955s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.646566731 (111h19m33.198037182s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.655804502 (111h19m33.207274963s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.678400780 (111h19m33.229871231s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:55.588855093 (111h19m54.140325554s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.058280180 (111h19m59.609750631s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.487675183 (111h20m0.039145654s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.488290434 (111h20m0.039760905s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection Nov 13 23:24:01.488833839 (111h20m0.0403043s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection Nov 13 23:24:01.489149691 (111h20m0.040620152s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection Nov 13 23:24:01.489463719 (111h20m0.04093418s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:01.489493796 (111h20m0.040964247s) [seg_pos={0 0}, upload=97944]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:24:01.748156531 (111h20m0.299627002s) [upload=97944]: Done uploading segments... Nov 13 23:24:01.748206497 (111h20m0.299676958s) [upload=97944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.261329204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................XX............X....X........X...X........X] duration: 30.262128971s trace: https://jaeger.dev.storj.io/trace/18a81ed93834dbe FAIL (2025-11-13 23:23:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== F42ENAIW4D3UEQ3TGHPONU6STXG3TXVIMYQKIA, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:55.884494570 (111h18m54.435965031s) [noise=true , address=145.79.227.30:4761 , piece_id=CLTFC6E56DXRRELZ6LCG7RC76S5OPZLNRJEYC7ZMTX5ZMJC7WFJQ, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:55.884939117 (111h18m54.436409578s) [noise=true , address=145.79.227.16:4910 , piece_id=6GACCFK7HALUEQIOHO5DYYU56ITBTATQAZ6G4G7Y42YMKA3VHZLQ, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:55.887285955 (111h18m54.438756417s) [noise=true , address=77.90.31.19:4843 , piece_id=YN5NNN3P7JQW3MLF7YUJLENLIYE6SEYVI7MRZIJYMI2OCAVFFKOA, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:55.888485290 (111h18m54.439955751s) [noise=true , address=145.79.227.17:4970 , piece_id=LT6OU7DYUF3WGK5CE4EQSKKBC66M43MM52OBXKAQDNDN77HU4IWA, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:55.890516227 (111h18m54.441986708s) [noise=true , address=145.79.227.22:4511 , piece_id=P6EEUSNFMMUTRINXOSKUT7RTFIROUPCDOCGBTHZ5T63OMDFZYMEQ, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:55.892296406 (111h18m54.443766857s) [noise=true , address=145.79.227.10:4401 , piece_id=UWPRD6U4PM7RAKC4GMWBAMVHGFWUUJZC3K2F3ZDSFSAE5TWEYBLQ, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:55.892998594 (111h18m54.444469045s) [noise=true , address=145.79.227.25:4617 , piece_id=HZVDL3BIYHIFKXELSHQ6JAWW2I4KJCFM7TD4MXHIR3WZUFRRPTPQ, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:55.894187127 (111h18m54.445657578s) [noise=true , address=145.79.227.32:4835 , piece_id=5QJC54A6BIUUAUORMVGUNYGTC5CEBWTKAHHUB2DDQ6JI7NP6TQCA, seg_pos={0 0}, upload=97933]: Uploading piece... Nov 13 23:22:57.578567361 (111h18m56.130037812s) [upload=97933]: Next returned nil segment Nov 13 23:22:57.578640540 (111h18m56.130111001s) [upload=97933]: Waiting for error group managing segments... Nov 13 23:22:57.978560737 (111h18m56.530031188s) [noise=true , address=77.90.31.21:4879 , piece_id=VM7CBFQO7YS35Q7IJASUC4U33XIUWNK4CNDM7AAUPBRHNGSKUECQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: <nil> Nov 13 23:22:57.988048965 (111h18m56.539519416s) [noise=true , address=77.90.31.19:4843 , piece_id=YN5NNN3P7JQW3MLF7YUJLENLIYE6SEYVI7MRZIJYMI2OCAVFFKOA, seg_pos={0 0}, upload=97933]: Done uploading piece. err: <nil> Nov 13 23:22:58.057475486 (111h18m56.608945947s) [noise=true , address=45.91.116.11:4117 , piece_id=TFJ2FBQNGUXG6C2UUO5QUA7YMMH3B3XKNJBWUGKHRZQ6RWI5VFCA, seg_pos={0 0}, upload=97933]: Done uploading piece. err: <nil> Nov 13 23:22:58.074620173 (111h18m56.626090634s) [noise=true , address=77.90.31.10:4520 , piece_id=RUGV7LTY2REOD6F42ENAIW4D3UEQ3TGHPONU6STXG3TXVIMYQKIA, seg_pos={0 0}, upload=97933]: Done uploading piece. err: <nil> Nov 13 23:23:24.915546808 (111h19m23.467017259s) [noise=true , address=145.79.227.17:4970 , piece_id=LT6OU7DYUF3WGK5CE4EQSKKBC66M43MM52OBXKAQDNDN77HU4IWA, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:59138->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:59138->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59138->145.79.227.17:4970: use of closed network connection Nov 13 23:23:24.915588257 (111h19m23.467058718s) [noise=true , address=145.79.227.16:4910 , piece_id=6GACCFK7HALUEQIOHO5DYYU56ITBTATQAZ6G4G7Y42YMKA3VHZLQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:60734->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:60734->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60734->145.79.227.16:4910: use of closed network connection Nov 13 23:23:24.915661216 (111h19m23.467131667s) [noise=true , address=145.79.227.10:4401 , piece_id=UWPRD6U4PM7RAKC4GMWBAMVHGFWUUJZC3K2F3ZDSFSAE5TWEYBLQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:33256->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:33256->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33256->145.79.227.10:4401: use of closed network connection Nov 13 23:23:24.915731890 (111h19m23.467202351s) [noise=true , address=145.79.227.22:4511 , piece_id=P6EEUSNFMMUTRINXOSKUT7RTFIROUPCDOCGBTHZ5T63OMDFZYMEQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:33428->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:33428->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33428->145.79.227.22:4511: use of closed network connection Nov 13 23:23:24.915767448 (111h19m23.467237899s) [noise=true , address=145.79.227.25:4617 , piece_id=HZVDL3BIYHIFKXELSHQ6JAWW2I4KJCFM7TD4MXHIR3WZUFRRPTPQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:46786->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:46786->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46786->145.79.227.25:4617: use of closed network connection Nov 13 23:23:24.916013316 (111h19m23.467483777s) [noise=true , address=145.79.227.11:4730 , piece_id=LLICR3UL2CMGWTWWDPYUSR5E7TFDEXV5HOZ4GGD7GR77Y2UTJPVQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:49966->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:49966->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49966->145.79.227.11:4730: use of closed network connection Nov 13 23:23:24.916070375 (111h19m23.467540836s) [noise=true , address=145.79.227.32:4835 , piece_id=5QJC54A6BIUUAUORMVGUNYGTC5CEBWTKAHHUB2DDQ6JI7NP6TQCA, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: read tcp 188.166.225.170:53734->145.79.227.32:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:23:24.916190694 (111h19m23.467661155s) [noise=true , address=145.79.227.9:4367 , piece_id=JRVOVLJLWD5A3QWBZ5KPCMSCEFE366SUJXUH3RY2IM34BJDKJDTQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:44080->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:44080->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44080->145.79.227.9:4367: use of closed network connection Nov 13 23:23:24.916473873 (111h19m23.467944334s) [noise=true , address=145.79.227.21:4475 , piece_id=ZT3COXAKYTBWS4I3YL7LYFUDISJC5PS6GPUTIFC2LIL45LFER2NQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:58398->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:23:24.916568453 (111h19m23.468038914s) [noise=true , address=145.79.227.30:4761 , piece_id=CLTFC6E56DXRRELZ6LCG7RC76S5OPZLNRJEYC7ZMTX5ZMJC7WFJQ, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:47538->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:23:24.916641542 (111h19m23.468112003s) [noise=true , address=145.79.227.20:4438 , piece_id=IZDZNV4QY3BMQ5W36HUP7UJS75Q5L3XNGT5AGTELBCSDPQ724SKA, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:44612->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:23:24.917097160 (111h19m23.468567611s) [noise=true , address=145.79.227.15:4875 , piece_id=DLJPGHBR74IIDTR4DBIA2KBOJTKDJNQJEFC3V6EE6FDQNW3ZP22Q, seg_pos={0 0}, upload=97933]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:50638->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:23:24.917130603 (111h19m23.468601064s) [seg_pos={0 0}, upload=97933]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 23:23:25.176176117 (111h19m23.727646588s) [upload=97933]: Done uploading segments... Nov 13 23:23:25.176222015 (111h19m23.727692476s) [upload=97933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.262153858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XX............X....X........X...X........] duration: 6.007836175s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:22:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.007858808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XX............X....X........X...X.......] duration: 6.093994714s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:21:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.094027034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XX............X....X........X...X......] duration: 5.986221733s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:21:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.986253903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XX............X....X........X...X.....] duration: 5.940009397s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:20:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.940069381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XX............X....X........X...X....] duration: 6.017718164s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:19:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.01774727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XX............X....X........X...X...] duration: 6.26423475s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:17:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.264272542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XX............X....X........X...X..] duration: 5.726966098s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:17:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.726990736s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XX............X....X........X...X.] duration: 24.289839321s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:16:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 24.289858958s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.279119618s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:16:49 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.279144596s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.210325971s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:16:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.210390304s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................XX............X....X........X...X] duration: 30.246720417s trace: https://jaeger.dev.storj.io/trace/43dc12f553ad92f5 FAIL (2025-11-13 23:16:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 15:57.364405992 (111h11m55.915876443s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.364658472 (111h11m55.916128923s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.365523690 (111h11m55.916994151s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.366638342 (111h11m55.918108794s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.368200317 (111h11m55.919670778s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369572960 (111h11m55.921043451s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.369790194 (111h11m55.921260645s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.371890363 (111h11m55.923360844s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:57.372867875 (111h11m55.924338336s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Uploading piece... Nov 13 23:15:59.139112967 (111h11m57.690583428s) [upload=97833]: Next returned nil segment Nov 13 23:15:59.139128787 (111h11m57.690599248s) [upload=97833]: Waiting for error group managing segments... Nov 13 23:15:59.458766160 (111h11m58.010236621s) [noise=true , address=45.91.116.9:4263 , piece_id=7237DZNO46KNYXAGRJZFCKWLHOWLB5ENQWKOJ524Z7A4XGE56OGQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.467189059 (111h11m58.01865951s) [noise=true , address=45.91.116.14:4190 , piece_id=IIKIHTGH3OS5JC45IDWEWTQUE22XJAFTAF4AK7AUIRQJBTZ2B3QA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.482433126 (111h11m58.033903587s) [noise=true , address=45.91.116.4:4940 , piece_id=JFGBOFGEMAQWHQ34XZGCQK43GGHXB56GTGW7EEYTW22EFLCVUCWQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.494886010 (111h11m58.046356471s) [noise=true , address=77.90.31.15:4698 , piece_id=DE27LDDF4LRQRC57S6PJDGLOSUZNEY7IIXHKCFUT7KP4WW7FJ6BA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.513074174 (111h11m58.064544635s) [noise=true , address=77.90.31.16:4735 , piece_id=HLE67VWXXV5Z7VS2RVGRO4UBNLV5KT7XPTBA32CPYQAF24MWY5YA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:15:59.536282456 (111h11m58.087752918s) [noise=true , address=77.90.31.21:4877 , piece_id=A2UM7FSV25YPKYVN3GZODWHWMSBK63ACVS7FYJLVHCW3WYTYR7NA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: <nil> Nov 13 23:16:26.395131974 (111h12m24.946602425s) [noise=true , address=145.79.227.20:4439 , piece_id=QXW7IBHAH7OGG7CMJXO3QSUWG6QG5B2HAUB4E4MS3YHTMOIVPQ5Q, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37034->145.79.227.20:4439: use of closed network connection Nov 13 23:16:26.395235741 (111h12m24.946706192s) [noise=true , address=145.79.227.17:4970 , piece_id=BG4GZCTAN6VZGIFBTEGOOXKDX7D5KQAOKCY6KZTKY5EQNNQQWXHA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55164->145.79.227.17:4970: use of closed network connection Nov 13 23:16:26.395311546 (111h12m24.946782007s) [noise=true , address=145.79.227.24:4582 , piece_id=7VKTWS7S3UDOKC7D75LDONLP6YBG5C3JJKBWXJ3L7QBIPC2K2WUA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43008->145.79.227.24:4582: use of closed network connection Nov 13 23:16:26.395384405 (111h12m24.946854856s) [noise=true , address=145.79.227.26:4655 , piece_id=OUOI7PLZVRKWG5G6MVZAHA5XAKAVHMQQ4J3SYNSOUUTH3TKJF2ZQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45662->145.79.227.26:4655: use of closed network connection Nov 13 23:16:26.395457514 (111h12m24.946927965s) [noise=true , address=145.79.227.18:4007 , piece_id=JBO244QPV3SWUUVD7VM6UON7THNDSF4QZB5O3MUQ7LI22DBXQVDA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35036->145.79.227.18:4007: use of closed network connection Nov 13 23:16:26.395527517 (111h12m24.946997978s) [noise=true , address=145.79.227.21:4475 , piece_id=KKJHZ3OZ2C5GK3EMUCCSXTYC7B6V7QB7PDE6XYMOVAUMVTRJGXKQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55102->145.79.227.21:4475: use of closed network connection Nov 13 23:16:26.395634822 (111h12m24.947105283s) [noise=true , address=145.79.227.30:4761 , piece_id=QNS75RYKULFJV6GYOHFDPN37BZ6PIT73TUVYI7P4ZPREUEXQQTLQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44770->145.79.227.30:4761: use of closed network connection Nov 13 23:16:26.395734071 (111h12m24.947204522s) [noise=true , address=145.79.227.22:4510 , piece_id=NXOTNQCTA76PAEADCSW7YQCAXNO7BCLTTIW23BRETQPNOPAIACXA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51652->145.79.227.22:4510: use of closed network connection Nov 13 23:16:26.396038129 (111h12m24.94750859s) [noise=true , address=145.79.227.10:4402 , piece_id=74OP4JRYX4HKP6WYF4IBBCWGVF65B2RDTF25ZI5YF5FJCAS7NVMQ, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47918->145.79.227.10:4402: use of closed network connection Nov 13 23:16:26.396204456 (111h12m24.947674917s) [noise=true , address=145.79.227.32:4836 , piece_id=FFGHAZG2PGHTXV5CKW7XO7AU7FVOJV32C6FRMO3E42MDSTEOS5OA, seg_pos={0 0}, upload=97833]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47796->145.79.227.32:4836: use of closed network connection Nov 13 23:16:26.396255143 (111h12m24.947725604s) [seg_pos={0 0}, upload=97833]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 23:16:26.639791564 (111h12m25.191262015s) [upload=97833]: Done uploading segments... Nov 13 23:16:26.639830839 (111h12m25.19130129s) [upload=97833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.246749562s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.160310617s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:15:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.160339903s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XX............X....X........X...] duration: 5.883962983s trace: https://jaeger.dev.storj.io/trace/459c383b1e5995dd FAIL (2025-11-13 23:15:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:12:42.900579739 (111h8m41.45205019s) [upload=97791]: Starting upload Nov 13 23:12:42.900643891 (111h8m41.452114353s) [upload=97791]: Done starting upload Nov 13 23:12:42.900725406 (111h8m41.452195877s) [upload=97791]: Uploading segments... Nov 13 23:12:42.900878889 (111h8m41.45234934s) [upload=97791]: Got next segment. Inline: false Nov 13 23:12:42.900924616 (111h8m41.452395067s) [upload=97791]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:12:42.900930908 (111h8m41.452401359s) [upload=97791]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:12:43.158728186 (111h8m41.710198637s) [seg_pos={0 0}, upload=97791]: Begin upload segment... Nov 13 23:12:43.158783230 (111h8m41.710253691s) [seg_pos={0 0}, upload=97791]: Done begin upload segment. Nov 13 23:12:43.158818577 (111h8m41.710289028s) [noise=true , address=145.79.227.9:4367 , piece_id=OXA7AM7W4VJCB4Q7PCWQTONHGGGAXWBFIT6EQFV4TMDVKK5NN2FQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.158851941 (111h8m41.710322392s) [noise=true , address=145.79.227.22:4510 , piece_id=VR7LBVHO42GO4RAI7CP3QLPBEISOLVFBXZLPZEW3XE32AB7GZNTQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159284194 (111h8m41.710754655s) [noise=true , address=145.79.227.13:4803 , piece_id=D56INZ5Z77RNA5KGFQZ4J22YKYSF4HHYTWKADRUZAPMXGA6Z6LOQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159448757 (111h8m41.710919218s) [noise=true , address=45.91.116.5:4298 , piece_id=EG222UOOYQAGOAFC4YOLKU6HL4VHAFEDJ43HP7ZZQ3SRCU7XPNPQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159839802 (111h8m41.711310263s) [noise=true , address=45.91.116.7:4009 , piece_id=ZNDDKLBJ5CKNHBWBS7TW3RF2EOV7EXYGGWTAZ7HJQOEK522EGTRQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.160704839 (111h8m41.71217529s) [noise=true , address=145.79.227.25:4619 , piece_id=5YIUS3XIVAOIQ5AFDVB5YXZDBPGKI4O5KHAQWNJ5PQOUPGBMGCCA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.160826030 (111h8m41.712296491s) [noise=true , address=145.79.227.21:4475 , piece_id=YDUM5SVKH54YXMWVEQQQ6ZTFHKZT2PADW3FT6RY7UYUM5CG2XMBQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.161627065 (111h8m41.713097516s) [noise=true , address=145.79.227.15:4875 , piece_id=7YUAMCER5O3PMTY2DTRDQZ3KVRHAPNJP25QQAKPC3IW2VFSDAN3A, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.161703030 (111h8m41.713173481s) [noise=true , address=145.79.227.16:4909 , piece_id=QYB4MRXZ2LHIYSGR4SZHS7ZPZTFVR2CRW5IPFLYFPOHZDVECXQ6A, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162233429 (111h8m41.713703891s) [noise=true , address=145.79.227.26:4653 , piece_id=ONFWOP3IGVW3GFHYF7CCOGNKUWWZ7UXM4SPEMHON7Y76J5CKNUVA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162690339 (111h8m41.7141608s) [noise=true , address=145.79.227.4:4186 , piece_id=LXRY2JZDOVJO3K5TOHXEJWZG73O65BTZ4DNK6ICBKLYNXKOIILPQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162982396 (111h8m41.714452847s) [noise=true , address=77.90.31.18:4807 , piece_id=O4NWG3LHHX3BXVBK2LNK2TYDO7EWIKVWODXZBYNIDT7BRBVQ7RFA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163038232 (111h8m41.714508693s) [noise=true , address=145.79.227.20:4437 , piece_id=ZQSBPMPMW3SGY6PP7YWQ2MMSAULGPECVT5LSYZSSUWYH5OOQETIQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163308016 (111h8m41.714778467s) [noise=true , address=145.79.227.27:4689 , piece_id=RLBDU56KC3TTDGIRL45DF4KTGWZ7PUALCPLZRG3PAW7JAO6YHAYQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163421702 (111h8m41.714892153s) [noise=true , address=145.79.227.8:4331 , piece_id=F5NKXOBDIZGGJQXN332FY6MR532FXII6WSKK2VGJPQKWA63B2WQA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.164046802 (111h8m41.715517263s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y566CN42BXTWBMDDJCUFIC76K23LYPKBHQHIH7USTPGUJR6U7QA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:44.869587276 (111h8m43.421057737s) [upload=97791]: Next returned nil segment Nov 13 23:12:44.869604979 (111h8m43.42107544s) [upload=97791]: Waiting for error group managing segments... Nov 13 23:12:45.207167934 (111h8m43.758638395s) [noise=true , address=45.91.116.5:4298 , piece_id=EG222UOOYQAGOAFC4YOLKU6HL4VHAFEDJ43HP7ZZQ3SRCU7XPNPQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.258085852 (111h8m43.809556313s) [noise=true , address=45.91.116.7:4009 , piece_id=ZNDDKLBJ5CKNHBWBS7TW3RF2EOV7EXYGGWTAZ7HJQOEK522EGTRQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.288021866 (111h8m43.839492327s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y566CN42BXTWBMDDJCUFIC76K23LYPKBHQHIH7USTPGUJR6U7QA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.385474143 (111h8m43.936944604s) [noise=true , address=77.90.31.18:4807 , piece_id=O4NWG3LHHX3BXVBK2LNK2TYDO7EWIKVWODXZBYNIDT7BRBVQ7RFA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.131815666 (111h8m49.683286117s) [noise=true , address=145.79.227.15:4875 , piece_id=7YUAMCER5O3PMTY2DTRDQZ3KVRHAPNJP25QQAKPC3IW2VFSDAN3A, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.438532885 (111h8m49.990003346s) [noise=true , address=145.79.227.8:4331 , piece_id=F5NKXOBDIZGGJQXN332FY6MR532FXII6WSKK2VGJPQKWA63B2WQA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.931109145 (111h8m50.482579606s) [noise=true , address=145.79.227.21:4475 , piece_id=YDUM5SVKH54YXMWVEQQQ6ZTFHKZT2PADW3FT6RY7UYUM5CG2XMBQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:54.059189419 (111h8m52.61065988s) [noise=true , address=145.79.227.16:4909 , piece_id=QYB4MRXZ2LHIYSGR4SZHS7ZPZTFVR2CRW5IPFLYFPOHZDVECXQ6A, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:59.993039877 (111h8m58.544510328s) [noise=true , address=145.79.227.4:4186 , piece_id=LXRY2JZDOVJO3K5TOHXEJWZG73O65BTZ4DNK6ICBKLYNXKOIILPQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:03.394457961 (111h9m1.945928422s) [noise=true , address=145.79.227.27:4689 , piece_id=RLBDU56KC3TTDGIRL45DF4KTGWZ7PUALCPLZRG3PAW7JAO6YHAYQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:10.522082014 (111h9m9.073552476s) [noise=true , address=145.79.227.13:4803 , piece_id=D56INZ5Z77RNA5KGFQZ4J22YKYSF4HHYTWKADRUZAPMXGA6Z6LOQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:11.805561005 (111h9m10.357031466s) [noise=true , address=145.79.227.9:4367 , piece_id=OXA7AM7W4VJCB4Q7PCWQTONHGGGAXWBFIT6EQFV4TMDVKK5NN2FQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:12.194560374 (111h9m10.746030855s) [noise=true , address=145.79.227.26:4653 , piece_id=ONFWOP3IGVW3GFHYF7CCOGNKUWWZ7UXM4SPEMHON7Y76J5CKNUVA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection Nov 13 23:13:12.194878880 (111h9m10.746349341s) [noise=true , address=145.79.227.20:4437 , piece_id=ZQSBPMPMW3SGY6PP7YWQ2MMSAULGPECVT5LSYZSSUWYH5OOQETIQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:13:12.195011132 (111h9m10.746481583s) [noise=true , address=145.79.227.25:4619 , piece_id=5YIUS3XIVAOIQ5AFDVB5YXZDBPGKI4O5KHAQWNJ5PQOUPGBMGCCA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection Nov 13 23:13:12.195214610 (111h9m10.746685081s) [noise=true , address=145.79.227.22:4510 , piece_id=VR7LBVHO42GO4RAI7CP3QLPBEISOLVFBXZLPZEW3XE32AB7GZNTQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection Nov 13 23:13:12.195259806 (111h9m10.746730257s) [seg_pos={0 0}, upload=97791]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:13:12.441272663 (111h9m10.992743134s) [upload=97791]: Done uploading segments... Nov 13 23:13:12.441342195 (111h9m10.992812656s) [upload=97791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.883994713s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.124896927s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:14:53 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.124915834s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XX............X....X........X..] duration: 5.915123638s trace: https://jaeger.dev.storj.io/trace/459c383b1e5995dd FAIL (2025-11-13 23:14:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:12:42.900579739 (111h8m41.45205019s) [upload=97791]: Starting upload Nov 13 23:12:42.900643891 (111h8m41.452114353s) [upload=97791]: Done starting upload Nov 13 23:12:42.900725406 (111h8m41.452195877s) [upload=97791]: Uploading segments... Nov 13 23:12:42.900878889 (111h8m41.45234934s) [upload=97791]: Got next segment. Inline: false Nov 13 23:12:42.900924616 (111h8m41.452395067s) [upload=97791]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:12:42.900930908 (111h8m41.452401359s) [upload=97791]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:12:43.158728186 (111h8m41.710198637s) [seg_pos={0 0}, upload=97791]: Begin upload segment... Nov 13 23:12:43.158783230 (111h8m41.710253691s) [seg_pos={0 0}, upload=97791]: Done begin upload segment. Nov 13 23:12:43.158818577 (111h8m41.710289028s) [noise=true , address=145.79.227.9:4367 , piece_id=OXA7AM7W4VJCB4Q7PCWQTONHGGGAXWBFIT6EQFV4TMDVKK5NN2FQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.158851941 (111h8m41.710322392s) [noise=true , address=145.79.227.22:4510 , piece_id=VR7LBVHO42GO4RAI7CP3QLPBEISOLVFBXZLPZEW3XE32AB7GZNTQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159284194 (111h8m41.710754655s) [noise=true , address=145.79.227.13:4803 , piece_id=D56INZ5Z77RNA5KGFQZ4J22YKYSF4HHYTWKADRUZAPMXGA6Z6LOQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159448757 (111h8m41.710919218s) [noise=true , address=45.91.116.5:4298 , piece_id=EG222UOOYQAGOAFC4YOLKU6HL4VHAFEDJ43HP7ZZQ3SRCU7XPNPQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159839802 (111h8m41.711310263s) [noise=true , address=45.91.116.7:4009 , piece_id=ZNDDKLBJ5CKNHBWBS7TW3RF2EOV7EXYGGWTAZ7HJQOEK522EGTRQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.160704839 (111h8m41.71217529s) [noise=true , address=145.79.227.25:4619 , piece_id=5YIUS3XIVAOIQ5AFDVB5YXZDBPGKI4O5KHAQWNJ5PQOUPGBMGCCA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.160826030 (111h8m41.712296491s) [noise=true , address=145.79.227.21:4475 , piece_id=YDUM5SVKH54YXMWVEQQQ6ZTFHKZT2PADW3FT6RY7UYUM5CG2XMBQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.161627065 (111h8m41.713097516s) [noise=true , address=145.79.227.15:4875 , piece_id=7YUAMCER5O3PMTY2DTRDQZ3KVRHAPNJP25QQAKPC3IW2VFSDAN3A, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.161703030 (111h8m41.713173481s) [noise=true , address=145.79.227.16:4909 , piece_id=QYB4MRXZ2LHIYSGR4SZHS7ZPZTFVR2CRW5IPFLYFPOHZDVECXQ6A, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162233429 (111h8m41.713703891s) [noise=true , address=145.79.227.26:4653 , piece_id=ONFWOP3IGVW3GFHYF7CCOGNKUWWZ7UXM4SPEMHON7Y76J5CKNUVA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162690339 (111h8m41.7141608s) [noise=true , address=145.79.227.4:4186 , piece_id=LXRY2JZDOVJO3K5TOHXEJWZG73O65BTZ4DNK6ICBKLYNXKOIILPQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162982396 (111h8m41.714452847s) [noise=true , address=77.90.31.18:4807 , piece_id=O4NWG3LHHX3BXVBK2LNK2TYDO7EWIKVWODXZBYNIDT7BRBVQ7RFA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163038232 (111h8m41.714508693s) [noise=true , address=145.79.227.20:4437 , piece_id=ZQSBPMPMW3SGY6PP7YWQ2MMSAULGPECVT5LSYZSSUWYH5OOQETIQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163308016 (111h8m41.714778467s) [noise=true , address=145.79.227.27:4689 , piece_id=RLBDU56KC3TTDGIRL45DF4KTGWZ7PUALCPLZRG3PAW7JAO6YHAYQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163421702 (111h8m41.714892153s) [noise=true , address=145.79.227.8:4331 , piece_id=F5NKXOBDIZGGJQXN332FY6MR532FXII6WSKK2VGJPQKWA63B2WQA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.164046802 (111h8m41.715517263s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y566CN42BXTWBMDDJCUFIC76K23LYPKBHQHIH7USTPGUJR6U7QA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:44.869587276 (111h8m43.421057737s) [upload=97791]: Next returned nil segment Nov 13 23:12:44.869604979 (111h8m43.42107544s) [upload=97791]: Waiting for error group managing segments... Nov 13 23:12:45.207167934 (111h8m43.758638395s) [noise=true , address=45.91.116.5:4298 , piece_id=EG222UOOYQAGOAFC4YOLKU6HL4VHAFEDJ43HP7ZZQ3SRCU7XPNPQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.258085852 (111h8m43.809556313s) [noise=true , address=45.91.116.7:4009 , piece_id=ZNDDKLBJ5CKNHBWBS7TW3RF2EOV7EXYGGWTAZ7HJQOEK522EGTRQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.288021866 (111h8m43.839492327s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y566CN42BXTWBMDDJCUFIC76K23LYPKBHQHIH7USTPGUJR6U7QA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.385474143 (111h8m43.936944604s) [noise=true , address=77.90.31.18:4807 , piece_id=O4NWG3LHHX3BXVBK2LNK2TYDO7EWIKVWODXZBYNIDT7BRBVQ7RFA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.131815666 (111h8m49.683286117s) [noise=true , address=145.79.227.15:4875 , piece_id=7YUAMCER5O3PMTY2DTRDQZ3KVRHAPNJP25QQAKPC3IW2VFSDAN3A, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.438532885 (111h8m49.990003346s) [noise=true , address=145.79.227.8:4331 , piece_id=F5NKXOBDIZGGJQXN332FY6MR532FXII6WSKK2VGJPQKWA63B2WQA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.931109145 (111h8m50.482579606s) [noise=true , address=145.79.227.21:4475 , piece_id=YDUM5SVKH54YXMWVEQQQ6ZTFHKZT2PADW3FT6RY7UYUM5CG2XMBQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:54.059189419 (111h8m52.61065988s) [noise=true , address=145.79.227.16:4909 , piece_id=QYB4MRXZ2LHIYSGR4SZHS7ZPZTFVR2CRW5IPFLYFPOHZDVECXQ6A, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:59.993039877 (111h8m58.544510328s) [noise=true , address=145.79.227.4:4186 , piece_id=LXRY2JZDOVJO3K5TOHXEJWZG73O65BTZ4DNK6ICBKLYNXKOIILPQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:03.394457961 (111h9m1.945928422s) [noise=true , address=145.79.227.27:4689 , piece_id=RLBDU56KC3TTDGIRL45DF4KTGWZ7PUALCPLZRG3PAW7JAO6YHAYQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:10.522082014 (111h9m9.073552476s) [noise=true , address=145.79.227.13:4803 , piece_id=D56INZ5Z77RNA5KGFQZ4J22YKYSF4HHYTWKADRUZAPMXGA6Z6LOQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:11.805561005 (111h9m10.357031466s) [noise=true , address=145.79.227.9:4367 , piece_id=OXA7AM7W4VJCB4Q7PCWQTONHGGGAXWBFIT6EQFV4TMDVKK5NN2FQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:12.194560374 (111h9m10.746030855s) [noise=true , address=145.79.227.26:4653 , piece_id=ONFWOP3IGVW3GFHYF7CCOGNKUWWZ7UXM4SPEMHON7Y76J5CKNUVA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection Nov 13 23:13:12.194878880 (111h9m10.746349341s) [noise=true , address=145.79.227.20:4437 , piece_id=ZQSBPMPMW3SGY6PP7YWQ2MMSAULGPECVT5LSYZSSUWYH5OOQETIQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:13:12.195011132 (111h9m10.746481583s) [noise=true , address=145.79.227.25:4619 , piece_id=5YIUS3XIVAOIQ5AFDVB5YXZDBPGKI4O5KHAQWNJ5PQOUPGBMGCCA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection Nov 13 23:13:12.195214610 (111h9m10.746685081s) [noise=true , address=145.79.227.22:4510 , piece_id=VR7LBVHO42GO4RAI7CP3QLPBEISOLVFBXZLPZEW3XE32AB7GZNTQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection Nov 13 23:13:12.195259806 (111h9m10.746730257s) [seg_pos={0 0}, upload=97791]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:13:12.441272663 (111h9m10.992743134s) [upload=97791]: Done uploading segments... Nov 13 23:13:12.441342195 (111h9m10.992812656s) [upload=97791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.915143977s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.413685017s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:14:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.413711587s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XX............X....X........X.] duration: 5.91845897s trace: https://jaeger.dev.storj.io/trace/459c383b1e5995dd FAIL (2025-11-13 23:13:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:12:42.900579739 (111h8m41.45205019s) [upload=97791]: Starting upload Nov 13 23:12:42.900643891 (111h8m41.452114353s) [upload=97791]: Done starting upload Nov 13 23:12:42.900725406 (111h8m41.452195877s) [upload=97791]: Uploading segments... Nov 13 23:12:42.900878889 (111h8m41.45234934s) [upload=97791]: Got next segment. Inline: false Nov 13 23:12:42.900924616 (111h8m41.452395067s) [upload=97791]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:12:42.900930908 (111h8m41.452401359s) [upload=97791]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:12:43.158728186 (111h8m41.710198637s) [seg_pos={0 0}, upload=97791]: Begin upload segment... Nov 13 23:12:43.158783230 (111h8m41.710253691s) [seg_pos={0 0}, upload=97791]: Done begin upload segment. Nov 13 23:12:43.158818577 (111h8m41.710289028s) [noise=true , address=145.79.227.9:4367 , piece_id=OXA7AM7W4VJCB4Q7PCWQTONHGGGAXWBFIT6EQFV4TMDVKK5NN2FQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.158851941 (111h8m41.710322392s) [noise=true , address=145.79.227.22:4510 , piece_id=VR7LBVHO42GO4RAI7CP3QLPBEISOLVFBXZLPZEW3XE32AB7GZNTQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159284194 (111h8m41.710754655s) [noise=true , address=145.79.227.13:4803 , piece_id=D56INZ5Z77RNA5KGFQZ4J22YKYSF4HHYTWKADRUZAPMXGA6Z6LOQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159448757 (111h8m41.710919218s) [noise=true , address=45.91.116.5:4298 , piece_id=EG222UOOYQAGOAFC4YOLKU6HL4VHAFEDJ43HP7ZZQ3SRCU7XPNPQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159839802 (111h8m41.711310263s) [noise=true , address=45.91.116.7:4009 , piece_id=ZNDDKLBJ5CKNHBWBS7TW3RF2EOV7EXYGGWTAZ7HJQOEK522EGTRQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.160704839 (111h8m41.71217529s) [noise=true , address=145.79.227.25:4619 , piece_id=5YIUS3XIVAOIQ5AFDVB5YXZDBPGKI4O5KHAQWNJ5PQOUPGBMGCCA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.160826030 (111h8m41.712296491s) [noise=true , address=145.79.227.21:4475 , piece_id=YDUM5SVKH54YXMWVEQQQ6ZTFHKZT2PADW3FT6RY7UYUM5CG2XMBQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.161627065 (111h8m41.713097516s) [noise=true , address=145.79.227.15:4875 , piece_id=7YUAMCER5O3PMTY2DTRDQZ3KVRHAPNJP25QQAKPC3IW2VFSDAN3A, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.161703030 (111h8m41.713173481s) [noise=true , address=145.79.227.16:4909 , piece_id=QYB4MRXZ2LHIYSGR4SZHS7ZPZTFVR2CRW5IPFLYFPOHZDVECXQ6A, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162233429 (111h8m41.713703891s) [noise=true , address=145.79.227.26:4653 , piece_id=ONFWOP3IGVW3GFHYF7CCOGNKUWWZ7UXM4SPEMHON7Y76J5CKNUVA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162690339 (111h8m41.7141608s) [noise=true , address=145.79.227.4:4186 , piece_id=LXRY2JZDOVJO3K5TOHXEJWZG73O65BTZ4DNK6ICBKLYNXKOIILPQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162982396 (111h8m41.714452847s) [noise=true , address=77.90.31.18:4807 , piece_id=O4NWG3LHHX3BXVBK2LNK2TYDO7EWIKVWODXZBYNIDT7BRBVQ7RFA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163038232 (111h8m41.714508693s) [noise=true , address=145.79.227.20:4437 , piece_id=ZQSBPMPMW3SGY6PP7YWQ2MMSAULGPECVT5LSYZSSUWYH5OOQETIQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163308016 (111h8m41.714778467s) [noise=true , address=145.79.227.27:4689 , piece_id=RLBDU56KC3TTDGIRL45DF4KTGWZ7PUALCPLZRG3PAW7JAO6YHAYQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163421702 (111h8m41.714892153s) [noise=true , address=145.79.227.8:4331 , piece_id=F5NKXOBDIZGGJQXN332FY6MR532FXII6WSKK2VGJPQKWA63B2WQA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.164046802 (111h8m41.715517263s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y566CN42BXTWBMDDJCUFIC76K23LYPKBHQHIH7USTPGUJR6U7QA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:44.869587276 (111h8m43.421057737s) [upload=97791]: Next returned nil segment Nov 13 23:12:44.869604979 (111h8m43.42107544s) [upload=97791]: Waiting for error group managing segments... Nov 13 23:12:45.207167934 (111h8m43.758638395s) [noise=true , address=45.91.116.5:4298 , piece_id=EG222UOOYQAGOAFC4YOLKU6HL4VHAFEDJ43HP7ZZQ3SRCU7XPNPQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.258085852 (111h8m43.809556313s) [noise=true , address=45.91.116.7:4009 , piece_id=ZNDDKLBJ5CKNHBWBS7TW3RF2EOV7EXYGGWTAZ7HJQOEK522EGTRQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.288021866 (111h8m43.839492327s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y566CN42BXTWBMDDJCUFIC76K23LYPKBHQHIH7USTPGUJR6U7QA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.385474143 (111h8m43.936944604s) [noise=true , address=77.90.31.18:4807 , piece_id=O4NWG3LHHX3BXVBK2LNK2TYDO7EWIKVWODXZBYNIDT7BRBVQ7RFA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.131815666 (111h8m49.683286117s) [noise=true , address=145.79.227.15:4875 , piece_id=7YUAMCER5O3PMTY2DTRDQZ3KVRHAPNJP25QQAKPC3IW2VFSDAN3A, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.438532885 (111h8m49.990003346s) [noise=true , address=145.79.227.8:4331 , piece_id=F5NKXOBDIZGGJQXN332FY6MR532FXII6WSKK2VGJPQKWA63B2WQA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.931109145 (111h8m50.482579606s) [noise=true , address=145.79.227.21:4475 , piece_id=YDUM5SVKH54YXMWVEQQQ6ZTFHKZT2PADW3FT6RY7UYUM5CG2XMBQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:54.059189419 (111h8m52.61065988s) [noise=true , address=145.79.227.16:4909 , piece_id=QYB4MRXZ2LHIYSGR4SZHS7ZPZTFVR2CRW5IPFLYFPOHZDVECXQ6A, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:59.993039877 (111h8m58.544510328s) [noise=true , address=145.79.227.4:4186 , piece_id=LXRY2JZDOVJO3K5TOHXEJWZG73O65BTZ4DNK6ICBKLYNXKOIILPQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:03.394457961 (111h9m1.945928422s) [noise=true , address=145.79.227.27:4689 , piece_id=RLBDU56KC3TTDGIRL45DF4KTGWZ7PUALCPLZRG3PAW7JAO6YHAYQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:10.522082014 (111h9m9.073552476s) [noise=true , address=145.79.227.13:4803 , piece_id=D56INZ5Z77RNA5KGFQZ4J22YKYSF4HHYTWKADRUZAPMXGA6Z6LOQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:11.805561005 (111h9m10.357031466s) [noise=true , address=145.79.227.9:4367 , piece_id=OXA7AM7W4VJCB4Q7PCWQTONHGGGAXWBFIT6EQFV4TMDVKK5NN2FQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:12.194560374 (111h9m10.746030855s) [noise=true , address=145.79.227.26:4653 , piece_id=ONFWOP3IGVW3GFHYF7CCOGNKUWWZ7UXM4SPEMHON7Y76J5CKNUVA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection Nov 13 23:13:12.194878880 (111h9m10.746349341s) [noise=true , address=145.79.227.20:4437 , piece_id=ZQSBPMPMW3SGY6PP7YWQ2MMSAULGPECVT5LSYZSSUWYH5OOQETIQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:13:12.195011132 (111h9m10.746481583s) [noise=true , address=145.79.227.25:4619 , piece_id=5YIUS3XIVAOIQ5AFDVB5YXZDBPGKI4O5KHAQWNJ5PQOUPGBMGCCA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection Nov 13 23:13:12.195214610 (111h9m10.746685081s) [noise=true , address=145.79.227.22:4510 , piece_id=VR7LBVHO42GO4RAI7CP3QLPBEISOLVFBXZLPZEW3XE32AB7GZNTQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection Nov 13 23:13:12.195259806 (111h9m10.746730257s) [seg_pos={0 0}, upload=97791]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:13:12.441272663 (111h9m10.992743134s) [upload=97791]: Done uploading segments... Nov 13 23:13:12.441342195 (111h9m10.992812656s) [upload=97791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.918485009s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.221687664s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:13:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.221716349s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................XX............X....X........X] duration: 30.249269669s trace: https://jaeger.dev.storj.io/trace/459c383b1e5995dd FAIL (2025-11-13 23:13:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:12:42.900579739 (111h8m41.45205019s) [upload=97791]: Starting upload Nov 13 23:12:42.900643891 (111h8m41.452114353s) [upload=97791]: Done starting upload Nov 13 23:12:42.900725406 (111h8m41.452195877s) [upload=97791]: Uploading segments... Nov 13 23:12:42.900878889 (111h8m41.45234934s) [upload=97791]: Got next segment. Inline: false Nov 13 23:12:42.900924616 (111h8m41.452395067s) [upload=97791]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:12:42.900930908 (111h8m41.452401359s) [upload=97791]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:12:43.158728186 (111h8m41.710198637s) [seg_pos={0 0}, upload=97791]: Begin upload segment... Nov 13 23:12:43.158783230 (111h8m41.710253691s) [seg_pos={0 0}, upload=97791]: Done begin upload segment. Nov 13 23:12:43.158818577 (111h8m41.710289028s) [noise=true , address=145.79.227.9:4367 , piece_id=OXA7AM7W4VJCB4Q7PCWQTONHGGGAXWBFIT6EQFV4TMDVKK5NN2FQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.158851941 (111h8m41.710322392s) [noise=true , address=145.79.227.22:4510 , piece_id=VR7LBVHO42GO4RAI7CP3QLPBEISOLVFBXZLPZEW3XE32AB7GZNTQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159284194 (111h8m41.710754655s) [noise=true , address=145.79.227.13:4803 , piece_id=D56INZ5Z77RNA5KGFQZ4J22YKYSF4HHYTWKADRUZAPMXGA6Z6LOQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159448757 (111h8m41.710919218s) [noise=true , address=45.91.116.5:4298 , piece_id=EG222UOOYQAGOAFC4YOLKU6HL4VHAFEDJ43HP7ZZQ3SRCU7XPNPQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.159839802 (111h8m41.711310263s) [noise=true , address=45.91.116.7:4009 , piece_id=ZNDDKLBJ5CKNHBWBS7TW3RF2EOV7EXYGGWTAZ7HJQOEK522EGTRQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.160704839 (111h8m41.71217529s) [noise=true , address=145.79.227.25:4619 , piece_id=5YIUS3XIVAOIQ5AFDVB5YXZDBPGKI4O5KHAQWNJ5PQOUPGBMGCCA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.160826030 (111h8m41.712296491s) [noise=true , address=145.79.227.21:4475 , piece_id=YDUM5SVKH54YXMWVEQQQ6ZTFHKZT2PADW3FT6RY7UYUM5CG2XMBQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.161627065 (111h8m41.713097516s) [noise=true , address=145.79.227.15:4875 , piece_id=7YUAMCER5O3PMTY2DTRDQZ3KVRHAPNJP25QQAKPC3IW2VFSDAN3A, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.161703030 (111h8m41.713173481s) [noise=true , address=145.79.227.16:4909 , piece_id=QYB4MRXZ2LHIYSGR4SZHS7ZPZTFVR2CRW5IPFLYFPOHZDVECXQ6A, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162233429 (111h8m41.713703891s) [noise=true , address=145.79.227.26:4653 , piece_id=ONFWOP3IGVW3GFHYF7CCOGNKUWWZ7UXM4SPEMHON7Y76J5CKNUVA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162690339 (111h8m41.7141608s) [noise=true , address=145.79.227.4:4186 , piece_id=LXRY2JZDOVJO3K5TOHXEJWZG73O65BTZ4DNK6ICBKLYNXKOIILPQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.162982396 (111h8m41.714452847s) [noise=true , address=77.90.31.18:4807 , piece_id=O4NWG3LHHX3BXVBK2LNK2TYDO7EWIKVWODXZBYNIDT7BRBVQ7RFA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163038232 (111h8m41.714508693s) [noise=true , address=145.79.227.20:4437 , piece_id=ZQSBPMPMW3SGY6PP7YWQ2MMSAULGPECVT5LSYZSSUWYH5OOQETIQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163308016 (111h8m41.714778467s) [noise=true , address=145.79.227.27:4689 , piece_id=RLBDU56KC3TTDGIRL45DF4KTGWZ7PUALCPLZRG3PAW7JAO6YHAYQ, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.163421702 (111h8m41.714892153s) [noise=true , address=145.79.227.8:4331 , piece_id=F5NKXOBDIZGGJQXN332FY6MR532FXII6WSKK2VGJPQKWA63B2WQA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:43.164046802 (111h8m41.715517263s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y566CN42BXTWBMDDJCUFIC76K23LYPKBHQHIH7USTPGUJR6U7QA, seg_pos={0 0}, upload=97791]: Uploading piece... Nov 13 23:12:44.869587276 (111h8m43.421057737s) [upload=97791]: Next returned nil segment Nov 13 23:12:44.869604979 (111h8m43.42107544s) [upload=97791]: Waiting for error group managing segments... Nov 13 23:12:45.207167934 (111h8m43.758638395s) [noise=true , address=45.91.116.5:4298 , piece_id=EG222UOOYQAGOAFC4YOLKU6HL4VHAFEDJ43HP7ZZQ3SRCU7XPNPQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.258085852 (111h8m43.809556313s) [noise=true , address=45.91.116.7:4009 , piece_id=ZNDDKLBJ5CKNHBWBS7TW3RF2EOV7EXYGGWTAZ7HJQOEK522EGTRQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.288021866 (111h8m43.839492327s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y566CN42BXTWBMDDJCUFIC76K23LYPKBHQHIH7USTPGUJR6U7QA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:45.385474143 (111h8m43.936944604s) [noise=true , address=77.90.31.18:4807 , piece_id=O4NWG3LHHX3BXVBK2LNK2TYDO7EWIKVWODXZBYNIDT7BRBVQ7RFA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.131815666 (111h8m49.683286117s) [noise=true , address=145.79.227.15:4875 , piece_id=7YUAMCER5O3PMTY2DTRDQZ3KVRHAPNJP25QQAKPC3IW2VFSDAN3A, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.438532885 (111h8m49.990003346s) [noise=true , address=145.79.227.8:4331 , piece_id=F5NKXOBDIZGGJQXN332FY6MR532FXII6WSKK2VGJPQKWA63B2WQA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:51.931109145 (111h8m50.482579606s) [noise=true , address=145.79.227.21:4475 , piece_id=YDUM5SVKH54YXMWVEQQQ6ZTFHKZT2PADW3FT6RY7UYUM5CG2XMBQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:54.059189419 (111h8m52.61065988s) [noise=true , address=145.79.227.16:4909 , piece_id=QYB4MRXZ2LHIYSGR4SZHS7ZPZTFVR2CRW5IPFLYFPOHZDVECXQ6A, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:12:59.993039877 (111h8m58.544510328s) [noise=true , address=145.79.227.4:4186 , piece_id=LXRY2JZDOVJO3K5TOHXEJWZG73O65BTZ4DNK6ICBKLYNXKOIILPQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:03.394457961 (111h9m1.945928422s) [noise=true , address=145.79.227.27:4689 , piece_id=RLBDU56KC3TTDGIRL45DF4KTGWZ7PUALCPLZRG3PAW7JAO6YHAYQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:10.522082014 (111h9m9.073552476s) [noise=true , address=145.79.227.13:4803 , piece_id=D56INZ5Z77RNA5KGFQZ4J22YKYSF4HHYTWKADRUZAPMXGA6Z6LOQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:11.805561005 (111h9m10.357031466s) [noise=true , address=145.79.227.9:4367 , piece_id=OXA7AM7W4VJCB4Q7PCWQTONHGGGAXWBFIT6EQFV4TMDVKK5NN2FQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: <nil> Nov 13 23:13:12.194560374 (111h9m10.746030855s) [noise=true , address=145.79.227.26:4653 , piece_id=ONFWOP3IGVW3GFHYF7CCOGNKUWWZ7UXM4SPEMHON7Y76J5CKNUVA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51574->145.79.227.26:4653: use of closed network connection Nov 13 23:13:12.194878880 (111h9m10.746349341s) [noise=true , address=145.79.227.20:4437 , piece_id=ZQSBPMPMW3SGY6PP7YWQ2MMSAULGPECVT5LSYZSSUWYH5OOQETIQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:13:12.195011132 (111h9m10.746481583s) [noise=true , address=145.79.227.25:4619 , piece_id=5YIUS3XIVAOIQ5AFDVB5YXZDBPGKI4O5KHAQWNJ5PQOUPGBMGCCA, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47706->145.79.227.25:4619: use of closed network connection Nov 13 23:13:12.195214610 (111h9m10.746685081s) [noise=true , address=145.79.227.22:4510 , piece_id=VR7LBVHO42GO4RAI7CP3QLPBEISOLVFBXZLPZEW3XE32AB7GZNTQ, seg_pos={0 0}, upload=97791]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49450->145.79.227.22:4510: use of closed network connection Nov 13 23:13:12.195259806 (111h9m10.746730257s) [seg_pos={0 0}, upload=97791]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:13:12.441272663 (111h9m10.992743134s) [upload=97791]: Done uploading segments... Nov 13 23:13:12.441342195 (111h9m10.992812656s) [upload=97791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.24929643s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.263398433s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:12:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.263427579s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XX............X....X........] duration: 5.96642306s trace: https://jaeger.dev.storj.io/trace/7a597331cd74105c FAIL (2025-11-13 23:11:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 13 23:06:01.786017718 (111h2m0.337488189s) [upload=97690]: Starting upload Nov 13 23:06:01.786112209 (111h2m0.33758267s) [upload=97690]: Done starting upload Nov 13 23:06:01.786321287 (111h2m0.337791758s) [upload=97690]: Uploading segments... Nov 13 23:06:01.786400347 (111h2m0.337870828s) [upload=97690]: Got next segment. Inline: false Nov 13 23:06:01.786444240 (111h2m0.337914701s) [upload=97690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:06:01.786450994 (111h2m0.337921455s) [upload=97690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:06:02.045368251 (111h2m0.596838722s) [seg_pos={0 0}, upload=97690]: Begin upload segment... Nov 13 23:06:02.045444366 (111h2m0.596914827s) [seg_pos={0 0}, upload=97690]: Done begin upload segment. Nov 13 23:06:02.045497568 (111h2m0.596968029s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.045630982 (111h2m0.597101463s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.046647998 (111h2m0.598118459s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047166876 (111h2m0.598637337s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047669533 (111h2m0.599139994s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048440271 (111h2m0.599910732s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048898173 (111h2m0.600368634s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.049356366 (111h2m0.600826837s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050066788 (111h2m0.601537249s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050566400 (111h2m0.602036861s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.054307234 (111h2m0.605777695s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.055200855 (111h2m0.606671316s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056033691 (111h2m0.607504152s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056686584 (111h2m0.608157045s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.057347713 (111h2m0.608818174s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.058057735 (111h2m0.609528196s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:03.766982075 (111h2m2.318452546s) [upload=97690]: Next returned nil segment Nov 13 23:06:03.767003416 (111h2m2.318473877s) [upload=97690]: Waiting for error group managing segments... Nov 13 23:06:04.093661125 (111h2m2.645131586s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.114432989 (111h2m2.66590346s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.147195895 (111h2m2.698666356s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.196607114 (111h2m2.748077575s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.207407540 (111h2m2.758878001s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.252190545 (111h2m2.803661006s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.292842424 (111h2m2.844312875s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:27.394277198 (111h2m25.945747649s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:29.722665255 (111h2m28.274135716s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.558038426 (111h2m29.109508877s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.663208820 (111h2m29.214679271s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:31.062805448 (111h2m29.614275909s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection Nov 13 23:06:31.062869480 (111h2m29.614339941s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection Nov 13 23:06:31.062914375 (111h2m29.614384826s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection Nov 13 23:06:31.063086212 (111h2m29.614556663s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 188.166.225.170:50042->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:06:31.063179480 (111h2m29.614649931s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection Nov 13 23:06:31.063204328 (111h2m29.614674789s) [seg_pos={0 0}, upload=97690]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 13 23:06:31.315724204 (111h2m29.867194655s) [upload=97690]: Done uploading segments... Nov 13 23:06:31.315757548 (111h2m29.867228009s) [upload=97690]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.966453188s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.26958097s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:11:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.269599926s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX............X....X.......] duration: 5.782885512s trace: https://jaeger.dev.storj.io/trace/7a597331cd74105c FAIL (2025-11-13 23:11:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 13 23:06:01.786017718 (111h2m0.337488189s) [upload=97690]: Starting upload Nov 13 23:06:01.786112209 (111h2m0.33758267s) [upload=97690]: Done starting upload Nov 13 23:06:01.786321287 (111h2m0.337791758s) [upload=97690]: Uploading segments... Nov 13 23:06:01.786400347 (111h2m0.337870828s) [upload=97690]: Got next segment. Inline: false Nov 13 23:06:01.786444240 (111h2m0.337914701s) [upload=97690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:06:01.786450994 (111h2m0.337921455s) [upload=97690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:06:02.045368251 (111h2m0.596838722s) [seg_pos={0 0}, upload=97690]: Begin upload segment... Nov 13 23:06:02.045444366 (111h2m0.596914827s) [seg_pos={0 0}, upload=97690]: Done begin upload segment. Nov 13 23:06:02.045497568 (111h2m0.596968029s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.045630982 (111h2m0.597101463s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.046647998 (111h2m0.598118459s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047166876 (111h2m0.598637337s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047669533 (111h2m0.599139994s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048440271 (111h2m0.599910732s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048898173 (111h2m0.600368634s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.049356366 (111h2m0.600826837s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050066788 (111h2m0.601537249s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050566400 (111h2m0.602036861s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.054307234 (111h2m0.605777695s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.055200855 (111h2m0.606671316s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056033691 (111h2m0.607504152s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056686584 (111h2m0.608157045s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.057347713 (111h2m0.608818174s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.058057735 (111h2m0.609528196s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:03.766982075 (111h2m2.318452546s) [upload=97690]: Next returned nil segment Nov 13 23:06:03.767003416 (111h2m2.318473877s) [upload=97690]: Waiting for error group managing segments... Nov 13 23:06:04.093661125 (111h2m2.645131586s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.114432989 (111h2m2.66590346s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.147195895 (111h2m2.698666356s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.196607114 (111h2m2.748077575s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.207407540 (111h2m2.758878001s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.252190545 (111h2m2.803661006s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.292842424 (111h2m2.844312875s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:27.394277198 (111h2m25.945747649s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:29.722665255 (111h2m28.274135716s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.558038426 (111h2m29.109508877s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.663208820 (111h2m29.214679271s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:31.062805448 (111h2m29.614275909s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection Nov 13 23:06:31.062869480 (111h2m29.614339941s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection Nov 13 23:06:31.062914375 (111h2m29.614384826s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection Nov 13 23:06:31.063086212 (111h2m29.614556663s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 188.166.225.170:50042->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:06:31.063179480 (111h2m29.614649931s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection Nov 13 23:06:31.063204328 (111h2m29.614674789s) [seg_pos={0 0}, upload=97690]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 13 23:06:31.315724204 (111h2m29.867194655s) [upload=97690]: Done uploading segments... Nov 13 23:06:31.315757548 (111h2m29.867228009s) [upload=97690]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.78291027s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.310276115s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:11:02 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.310302094s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX............X....X......] duration: 5.997889868s trace: https://jaeger.dev.storj.io/trace/7a597331cd74105c FAIL (2025-11-13 23:10:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 13 23:06:01.786017718 (111h2m0.337488189s) [upload=97690]: Starting upload Nov 13 23:06:01.786112209 (111h2m0.33758267s) [upload=97690]: Done starting upload Nov 13 23:06:01.786321287 (111h2m0.337791758s) [upload=97690]: Uploading segments... Nov 13 23:06:01.786400347 (111h2m0.337870828s) [upload=97690]: Got next segment. Inline: false Nov 13 23:06:01.786444240 (111h2m0.337914701s) [upload=97690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:06:01.786450994 (111h2m0.337921455s) [upload=97690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:06:02.045368251 (111h2m0.596838722s) [seg_pos={0 0}, upload=97690]: Begin upload segment... Nov 13 23:06:02.045444366 (111h2m0.596914827s) [seg_pos={0 0}, upload=97690]: Done begin upload segment. Nov 13 23:06:02.045497568 (111h2m0.596968029s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.045630982 (111h2m0.597101463s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.046647998 (111h2m0.598118459s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047166876 (111h2m0.598637337s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047669533 (111h2m0.599139994s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048440271 (111h2m0.599910732s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048898173 (111h2m0.600368634s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.049356366 (111h2m0.600826837s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050066788 (111h2m0.601537249s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050566400 (111h2m0.602036861s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.054307234 (111h2m0.605777695s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.055200855 (111h2m0.606671316s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056033691 (111h2m0.607504152s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056686584 (111h2m0.608157045s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.057347713 (111h2m0.608818174s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.058057735 (111h2m0.609528196s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:03.766982075 (111h2m2.318452546s) [upload=97690]: Next returned nil segment Nov 13 23:06:03.767003416 (111h2m2.318473877s) [upload=97690]: Waiting for error group managing segments... Nov 13 23:06:04.093661125 (111h2m2.645131586s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.114432989 (111h2m2.66590346s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.147195895 (111h2m2.698666356s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.196607114 (111h2m2.748077575s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.207407540 (111h2m2.758878001s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.252190545 (111h2m2.803661006s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.292842424 (111h2m2.844312875s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:27.394277198 (111h2m25.945747649s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:29.722665255 (111h2m28.274135716s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.558038426 (111h2m29.109508877s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.663208820 (111h2m29.214679271s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:31.062805448 (111h2m29.614275909s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection Nov 13 23:06:31.062869480 (111h2m29.614339941s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection Nov 13 23:06:31.062914375 (111h2m29.614384826s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection Nov 13 23:06:31.063086212 (111h2m29.614556663s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 188.166.225.170:50042->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:06:31.063179480 (111h2m29.614649931s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection Nov 13 23:06:31.063204328 (111h2m29.614674789s) [seg_pos={0 0}, upload=97690]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 13 23:06:31.315724204 (111h2m29.867194655s) [upload=97690]: Done uploading segments... Nov 13 23:06:31.315757548 (111h2m29.867228009s) [upload=97690]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.997919956s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.511285521s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:09:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.511308783s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX............X....X.....] duration: 5.954434331s trace: https://jaeger.dev.storj.io/trace/7a597331cd74105c FAIL (2025-11-13 23:09:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 13 23:06:01.786017718 (111h2m0.337488189s) [upload=97690]: Starting upload Nov 13 23:06:01.786112209 (111h2m0.33758267s) [upload=97690]: Done starting upload Nov 13 23:06:01.786321287 (111h2m0.337791758s) [upload=97690]: Uploading segments... Nov 13 23:06:01.786400347 (111h2m0.337870828s) [upload=97690]: Got next segment. Inline: false Nov 13 23:06:01.786444240 (111h2m0.337914701s) [upload=97690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:06:01.786450994 (111h2m0.337921455s) [upload=97690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:06:02.045368251 (111h2m0.596838722s) [seg_pos={0 0}, upload=97690]: Begin upload segment... Nov 13 23:06:02.045444366 (111h2m0.596914827s) [seg_pos={0 0}, upload=97690]: Done begin upload segment. Nov 13 23:06:02.045497568 (111h2m0.596968029s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.045630982 (111h2m0.597101463s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.046647998 (111h2m0.598118459s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047166876 (111h2m0.598637337s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047669533 (111h2m0.599139994s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048440271 (111h2m0.599910732s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048898173 (111h2m0.600368634s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.049356366 (111h2m0.600826837s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050066788 (111h2m0.601537249s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050566400 (111h2m0.602036861s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.054307234 (111h2m0.605777695s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.055200855 (111h2m0.606671316s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056033691 (111h2m0.607504152s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056686584 (111h2m0.608157045s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.057347713 (111h2m0.608818174s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.058057735 (111h2m0.609528196s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:03.766982075 (111h2m2.318452546s) [upload=97690]: Next returned nil segment Nov 13 23:06:03.767003416 (111h2m2.318473877s) [upload=97690]: Waiting for error group managing segments... Nov 13 23:06:04.093661125 (111h2m2.645131586s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.114432989 (111h2m2.66590346s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.147195895 (111h2m2.698666356s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.196607114 (111h2m2.748077575s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.207407540 (111h2m2.758878001s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.252190545 (111h2m2.803661006s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.292842424 (111h2m2.844312875s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:27.394277198 (111h2m25.945747649s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:29.722665255 (111h2m28.274135716s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.558038426 (111h2m29.109508877s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.663208820 (111h2m29.214679271s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:31.062805448 (111h2m29.614275909s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection Nov 13 23:06:31.062869480 (111h2m29.614339941s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection Nov 13 23:06:31.062914375 (111h2m29.614384826s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection Nov 13 23:06:31.063086212 (111h2m29.614556663s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 188.166.225.170:50042->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:06:31.063179480 (111h2m29.614649931s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection Nov 13 23:06:31.063204328 (111h2m29.614674789s) [seg_pos={0 0}, upload=97690]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 13 23:06:31.315724204 (111h2m29.867194655s) [upload=97690]: Done uploading segments... Nov 13 23:06:31.315757548 (111h2m29.867228009s) [upload=97690]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.954458377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XX............X....X....] duration: 5.959033028s trace: https://jaeger.dev.storj.io/trace/7a597331cd74105c FAIL (2025-11-13 23:09:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 13 23:06:01.786017718 (111h2m0.337488189s) [upload=97690]: Starting upload Nov 13 23:06:01.786112209 (111h2m0.33758267s) [upload=97690]: Done starting upload Nov 13 23:06:01.786321287 (111h2m0.337791758s) [upload=97690]: Uploading segments... Nov 13 23:06:01.786400347 (111h2m0.337870828s) [upload=97690]: Got next segment. Inline: false Nov 13 23:06:01.786444240 (111h2m0.337914701s) [upload=97690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:06:01.786450994 (111h2m0.337921455s) [upload=97690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:06:02.045368251 (111h2m0.596838722s) [seg_pos={0 0}, upload=97690]: Begin upload segment... Nov 13 23:06:02.045444366 (111h2m0.596914827s) [seg_pos={0 0}, upload=97690]: Done begin upload segment. Nov 13 23:06:02.045497568 (111h2m0.596968029s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.045630982 (111h2m0.597101463s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.046647998 (111h2m0.598118459s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047166876 (111h2m0.598637337s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047669533 (111h2m0.599139994s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048440271 (111h2m0.599910732s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048898173 (111h2m0.600368634s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.049356366 (111h2m0.600826837s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050066788 (111h2m0.601537249s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050566400 (111h2m0.602036861s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.054307234 (111h2m0.605777695s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.055200855 (111h2m0.606671316s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056033691 (111h2m0.607504152s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056686584 (111h2m0.608157045s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.057347713 (111h2m0.608818174s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.058057735 (111h2m0.609528196s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:03.766982075 (111h2m2.318452546s) [upload=97690]: Next returned nil segment Nov 13 23:06:03.767003416 (111h2m2.318473877s) [upload=97690]: Waiting for error group managing segments... Nov 13 23:06:04.093661125 (111h2m2.645131586s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.114432989 (111h2m2.66590346s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.147195895 (111h2m2.698666356s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.196607114 (111h2m2.748077575s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.207407540 (111h2m2.758878001s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.252190545 (111h2m2.803661006s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.292842424 (111h2m2.844312875s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:27.394277198 (111h2m25.945747649s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:29.722665255 (111h2m28.274135716s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.558038426 (111h2m29.109508877s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.663208820 (111h2m29.214679271s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:31.062805448 (111h2m29.614275909s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection Nov 13 23:06:31.062869480 (111h2m29.614339941s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection Nov 13 23:06:31.062914375 (111h2m29.614384826s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection Nov 13 23:06:31.063086212 (111h2m29.614556663s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 188.166.225.170:50042->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:06:31.063179480 (111h2m29.614649931s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection Nov 13 23:06:31.063204328 (111h2m29.614674789s) [seg_pos={0 0}, upload=97690]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 13 23:06:31.315724204 (111h2m29.867194655s) [upload=97690]: Done uploading segments... Nov 13 23:06:31.315757548 (111h2m29.867228009s) [upload=97690]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.959063928s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.11959542s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:08:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.119620827s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XX............X....X...] duration: 5.831223755s trace: https://jaeger.dev.storj.io/trace/7a597331cd74105c FAIL (2025-11-13 23:08:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 13 23:06:01.786017718 (111h2m0.337488189s) [upload=97690]: Starting upload Nov 13 23:06:01.786112209 (111h2m0.33758267s) [upload=97690]: Done starting upload Nov 13 23:06:01.786321287 (111h2m0.337791758s) [upload=97690]: Uploading segments... Nov 13 23:06:01.786400347 (111h2m0.337870828s) [upload=97690]: Got next segment. Inline: false Nov 13 23:06:01.786444240 (111h2m0.337914701s) [upload=97690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:06:01.786450994 (111h2m0.337921455s) [upload=97690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:06:02.045368251 (111h2m0.596838722s) [seg_pos={0 0}, upload=97690]: Begin upload segment... Nov 13 23:06:02.045444366 (111h2m0.596914827s) [seg_pos={0 0}, upload=97690]: Done begin upload segment. Nov 13 23:06:02.045497568 (111h2m0.596968029s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.045630982 (111h2m0.597101463s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.046647998 (111h2m0.598118459s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047166876 (111h2m0.598637337s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047669533 (111h2m0.599139994s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048440271 (111h2m0.599910732s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048898173 (111h2m0.600368634s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.049356366 (111h2m0.600826837s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050066788 (111h2m0.601537249s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050566400 (111h2m0.602036861s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.054307234 (111h2m0.605777695s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.055200855 (111h2m0.606671316s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056033691 (111h2m0.607504152s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056686584 (111h2m0.608157045s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.057347713 (111h2m0.608818174s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.058057735 (111h2m0.609528196s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:03.766982075 (111h2m2.318452546s) [upload=97690]: Next returned nil segment Nov 13 23:06:03.767003416 (111h2m2.318473877s) [upload=97690]: Waiting for error group managing segments... Nov 13 23:06:04.093661125 (111h2m2.645131586s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.114432989 (111h2m2.66590346s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.147195895 (111h2m2.698666356s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.196607114 (111h2m2.748077575s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.207407540 (111h2m2.758878001s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.252190545 (111h2m2.803661006s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.292842424 (111h2m2.844312875s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:27.394277198 (111h2m25.945747649s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:29.722665255 (111h2m28.274135716s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.558038426 (111h2m29.109508877s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.663208820 (111h2m29.214679271s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:31.062805448 (111h2m29.614275909s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection Nov 13 23:06:31.062869480 (111h2m29.614339941s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection Nov 13 23:06:31.062914375 (111h2m29.614384826s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection Nov 13 23:06:31.063086212 (111h2m29.614556663s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 188.166.225.170:50042->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:06:31.063179480 (111h2m29.614649931s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection Nov 13 23:06:31.063204328 (111h2m29.614674789s) [seg_pos={0 0}, upload=97690]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 13 23:06:31.315724204 (111h2m29.867194655s) [upload=97690]: Done uploading segments... Nov 13 23:06:31.315757548 (111h2m29.867228009s) [upload=97690]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.831279191s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.346651607s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:08:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.346719577s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.258402745s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:08:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.258430798s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XX............X....X..] duration: 5.938606421s trace: https://jaeger.dev.storj.io/trace/7a597331cd74105c FAIL (2025-11-13 23:07:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 13 23:06:01.786017718 (111h2m0.337488189s) [upload=97690]: Starting upload Nov 13 23:06:01.786112209 (111h2m0.33758267s) [upload=97690]: Done starting upload Nov 13 23:06:01.786321287 (111h2m0.337791758s) [upload=97690]: Uploading segments... Nov 13 23:06:01.786400347 (111h2m0.337870828s) [upload=97690]: Got next segment. Inline: false Nov 13 23:06:01.786444240 (111h2m0.337914701s) [upload=97690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:06:01.786450994 (111h2m0.337921455s) [upload=97690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:06:02.045368251 (111h2m0.596838722s) [seg_pos={0 0}, upload=97690]: Begin upload segment... Nov 13 23:06:02.045444366 (111h2m0.596914827s) [seg_pos={0 0}, upload=97690]: Done begin upload segment. Nov 13 23:06:02.045497568 (111h2m0.596968029s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.045630982 (111h2m0.597101463s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.046647998 (111h2m0.598118459s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047166876 (111h2m0.598637337s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047669533 (111h2m0.599139994s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048440271 (111h2m0.599910732s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048898173 (111h2m0.600368634s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.049356366 (111h2m0.600826837s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050066788 (111h2m0.601537249s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050566400 (111h2m0.602036861s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.054307234 (111h2m0.605777695s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.055200855 (111h2m0.606671316s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056033691 (111h2m0.607504152s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056686584 (111h2m0.608157045s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.057347713 (111h2m0.608818174s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.058057735 (111h2m0.609528196s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:03.766982075 (111h2m2.318452546s) [upload=97690]: Next returned nil segment Nov 13 23:06:03.767003416 (111h2m2.318473877s) [upload=97690]: Waiting for error group managing segments... Nov 13 23:06:04.093661125 (111h2m2.645131586s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.114432989 (111h2m2.66590346s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.147195895 (111h2m2.698666356s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.196607114 (111h2m2.748077575s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.207407540 (111h2m2.758878001s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.252190545 (111h2m2.803661006s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.292842424 (111h2m2.844312875s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:27.394277198 (111h2m25.945747649s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:29.722665255 (111h2m28.274135716s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.558038426 (111h2m29.109508877s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.663208820 (111h2m29.214679271s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:31.062805448 (111h2m29.614275909s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection Nov 13 23:06:31.062869480 (111h2m29.614339941s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection Nov 13 23:06:31.062914375 (111h2m29.614384826s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection Nov 13 23:06:31.063086212 (111h2m29.614556663s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 188.166.225.170:50042->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:06:31.063179480 (111h2m29.614649931s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection Nov 13 23:06:31.063204328 (111h2m29.614674789s) [seg_pos={0 0}, upload=97690]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 13 23:06:31.315724204 (111h2m29.867194655s) [upload=97690]: Done uploading segments... Nov 13 23:06:31.315757548 (111h2m29.867228009s) [upload=97690]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.938636838s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.28378513s trace: https://jaeger.dev.storj.io/trace/17b8d02cc19b4555 FAIL (2025-11-13 23:07:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:43622->45.38.24.2:443: read: connection reset by peer Duration: 6.283811672s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XX............X....X.] duration: 5.834692603s trace: https://jaeger.dev.storj.io/trace/7a597331cd74105c FAIL (2025-11-13 23:06:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 13 23:06:01.786017718 (111h2m0.337488189s) [upload=97690]: Starting upload Nov 13 23:06:01.786112209 (111h2m0.33758267s) [upload=97690]: Done starting upload Nov 13 23:06:01.786321287 (111h2m0.337791758s) [upload=97690]: Uploading segments... Nov 13 23:06:01.786400347 (111h2m0.337870828s) [upload=97690]: Got next segment. Inline: false Nov 13 23:06:01.786444240 (111h2m0.337914701s) [upload=97690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:06:01.786450994 (111h2m0.337921455s) [upload=97690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:06:02.045368251 (111h2m0.596838722s) [seg_pos={0 0}, upload=97690]: Begin upload segment... Nov 13 23:06:02.045444366 (111h2m0.596914827s) [seg_pos={0 0}, upload=97690]: Done begin upload segment. Nov 13 23:06:02.045497568 (111h2m0.596968029s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.045630982 (111h2m0.597101463s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.046647998 (111h2m0.598118459s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047166876 (111h2m0.598637337s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.047669533 (111h2m0.599139994s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048440271 (111h2m0.599910732s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.048898173 (111h2m0.600368634s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.049356366 (111h2m0.600826837s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050066788 (111h2m0.601537249s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.050566400 (111h2m0.602036861s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.054307234 (111h2m0.605777695s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.055200855 (111h2m0.606671316s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056033691 (111h2m0.607504152s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.056686584 (111h2m0.608157045s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.057347713 (111h2m0.608818174s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:02.058057735 (111h2m0.609528196s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Uploading piece... Nov 13 23:06:03.766982075 (111h2m2.318452546s) [upload=97690]: Next returned nil segment Nov 13 23:06:03.767003416 (111h2m2.318473877s) [upload=97690]: Waiting for error group managing segments... Nov 13 23:06:04.093661125 (111h2m2.645131586s) [noise=true , address=77.90.31.11:4555 , piece_id=L6O2EMSVXSOMMWLFJI4P6YIFLMQNGFWTAKF5PHMTL3GABHUBQQRQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.114432989 (111h2m2.66590346s) [noise=true , address=77.90.31.18:4807 , piece_id=HPK54S7QJVWUAH7JSYYAPA7JZJDK6XJJBVDB2V6VKOMEFEGUXHHA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.147195895 (111h2m2.698666356s) [noise=true , address=77.90.31.18:4805 , piece_id=XYNFG3ZXSMSE7UNULCZ24YPJSBONBNHS4O2BRLRA227BTYWPNAMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.196607114 (111h2m2.748077575s) [noise=true , address=77.90.31.12:4589 , piece_id=O2YQ2Z74GAEWJF5QQTAKFACERAWBL5OFRC3GQJYSVO5Z44VQGSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.207407540 (111h2m2.758878001s) [noise=true , address=45.91.116.6:4974 , piece_id=6GJMNBCAK7U75SDWDG27V7OXWS6NLHH53OJSIL4BR2N3CPHUKOQQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.252190545 (111h2m2.803661006s) [noise=true , address=77.90.31.19:4841 , piece_id=CP5B77A43KKQPRM4NJXF64IV72VX2XLXEUTSKKUOKQVGFBTNCSCQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:04.292842424 (111h2m2.844312875s) [noise=true , address=77.90.31.17:4772 , piece_id=IQQA4QBTEGP2YRQKFV477HE25SEF4Z2RKTCE2RBSR64UDMRAAXLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:27.394277198 (111h2m25.945747649s) [noise=true , address=145.79.227.23:4545 , piece_id=5POZUARV6DY3VGGQXAAUNIY4QKTK4GBRT4PUJUW7GBOBVBSQNW2A, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:29.722665255 (111h2m28.274135716s) [noise=true , address=145.79.227.31:4799 , piece_id=OM4YTL6ZUW7AW6S4CAZDVZS443AQXLCVJWDU5IMGZ373Z3NOWFLA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.558038426 (111h2m29.109508877s) [noise=true , address=145.79.227.4:4186 , piece_id=K3B2BPNV7PHG6INZVEUGJ3QLNWLMXVHVBZWCHMVGEEFYULIBERMQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:30.663208820 (111h2m29.214679271s) [noise=true , address=145.79.227.30:4761 , piece_id=6NS3EGW3IXN6HQ4DR66ULCOWB54U7Q656HAZQ5GDBAPKIFVYISFA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: <nil> Nov 13 23:06:31.062805448 (111h2m29.614275909s) [noise=true , address=145.79.227.21:4474 , piece_id=GM2BIYDO5JLPLL7JXVROGAAJN4DBH4ZDYSK76FC7CTKGRLIPJCBQ, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43946->145.79.227.21:4474: use of closed network connection Nov 13 23:06:31.062869480 (111h2m29.614339941s) [noise=true , address=145.79.227.20:4438 , piece_id=HFHNCTNU2OFLUMK3DHLBZHC3Y64RH727IEEFFC7QCQEMQ4G6CM4Q, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54224->145.79.227.20:4438: use of closed network connection Nov 13 23:06:31.062914375 (111h2m29.614384826s) [noise=true , address=145.79.227.10:4401 , piece_id=QPIKIBAHAB6ZWQB5UZDMACTPFH64NMKSXP34UO6WJNKWPUN3COEA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43292->145.79.227.10:4401: use of closed network connection Nov 13 23:06:31.063086212 (111h2m29.614556663s) [noise=true , address=145.79.227.15:4876 , piece_id=VH7TLRIX35WRA3P2RK5OYHKUV5S5ITHQBBWHHUFCBJTDNRY7TZNA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 188.166.225.170:50042->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:06:31.063179480 (111h2m29.614649931s) [noise=true , address=145.79.227.8:4331 , piece_id=XO4SCGS657IXOE4UN7QRINLXSEF3B7OURT752BIKVPSLFDCCKKWA, seg_pos={0 0}, upload=97690]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55038->145.79.227.8:4331: use of closed network connection Nov 13 23:06:31.063204328 (111h2m29.614674789s) [seg_pos={0 0}, upload=97690]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 13 23:06:31.315724204 (111h2m29.867194655s) [upload=97690]: Done uploading segments... Nov 13 23:06:31.315757548 (111h2m29.867228009s) [upload=97690]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.834709875s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)