Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.741545053s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:03:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.741563437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 10.698319979s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:03:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.698345398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.768252086s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:03:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.768273858s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.......X..........................X............X..........X.] duration: 4.076491291s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:03:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 4.076504407s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.719158126s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:03:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.719178625s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [........X..........................X............X..........X] duration: 1m0.002399669s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:03:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 1m0.002426772s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 4.717203626s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:02:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.717227884s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.700793768s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:02:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.700812694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 9.581234676s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:02:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.581255415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.683086489s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:02:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.683109825s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.684844433s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:01:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.684866033s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 8.835497685s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:01:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.835520248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.759565106s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:01:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.75958684s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.67475388s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:01:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.674769068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.242599456s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:01:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.242622849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.781639885s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:01:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.781660735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........................X............X..........] duration: 4.236936913s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 04:01:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 4.236960309s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.756613952s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:00:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.75662907s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........................X............X.........] duration: 4.371307137s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 04:00:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 4.371328216s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.725560975s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:00:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.725607425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.622257255s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:00:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 6.622279828s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.541065466s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:00:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.541086116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.733187427s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:00:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.73321544s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.947137936s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 03:59:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.947160309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 16.626489292s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:59:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.626511845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........................X............X........] duration: 4.391215953s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:59:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 4.391237013s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.067657309s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:59:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 6.0676738s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.258285516s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:59:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.258312388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 7.661098917s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:59:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.661123704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 4.748134133s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:58:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.748159392s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.805369391s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 03:58:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.805390161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 10.722809311s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:58:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.722826524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [............X..........................X............X.......] duration: 3.96780825s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:58:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 3.967828908s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.706370055s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:58:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.706389933s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.694305474s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:58:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.69432941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.637305196s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:57:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.637328731s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........................X............X......] duration: 7.569527917s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:57:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 7.569546923s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.796607222s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 03:57:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.796654882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 4.724264289s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:57:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.724284507s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.224404269s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:57:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.224418205s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 4.008582085s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 03:57:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.008606082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.771394462s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:57:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.77141489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........................X............X.....] duration: 36.900404668s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:57:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 36.900420649s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.821718399s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:56:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.821740512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 4.657984585s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:56:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.658003601s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.735971319s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 03:56:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.735990585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [...............X..........................X............X....] duration: 6.365397487s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:56:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 6.365424249s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.698315668s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:56:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.698335274s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.728296364s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:55:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.728316722s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 15.36023023s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:55:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.360251922s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)