Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 9.868572384s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:15:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.868581442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 19.356430047s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:14:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.356453962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [....................X............X..........X...............] duration: 3.98517201s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:14:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 3.985191849s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.733756581s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:14:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.73377659s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.777406205s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:14:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.777436823s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 4.720566482s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:14:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.720590379s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 8.688920141s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:14:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.68893465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.761092215s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:14:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.761125379s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.892104334s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:14:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.892126457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.188063982s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:14:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 5.188087004s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.720170075s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:13:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.720190653s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.....................X............X..........X..............] duration: 3.986299427s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:13:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 3.986323995s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.766743051s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:13:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.766766798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.717032441s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.717054513s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [......................X............X..........X.............] duration: 4.175771933s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:13:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 4.175795417s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 15.832732329s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:13:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.832756125s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 4.674914238s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:12:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.674929608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.685126774s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:12:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.685140159s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.096990465s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:12:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.097011445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 9.819601366s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:12:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.819634579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.......................X............X..........X............] duration: 4.017110679s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:12:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 4.017124004s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.741796353s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:11:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.741850206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.141441397s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:11:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 5.141462587s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.706445121s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:11:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.706468636s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [........................X............X..........X...........] duration: 3.964728636s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:11:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 3.964748614s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.868906754s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:11:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.868932264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 8.392542346s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:11:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.392569256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.814382025s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:10:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.81440581s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.732051085s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:10:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.732071795s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 18.516985091s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:10:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.517009727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.........................X............X..........X..........] duration: 4.097267167s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:10:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 4.097280463s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.793407074s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:10:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.793419779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.983074736s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:09:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.983110334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 13.535669188s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:09:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.535689737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.862932934s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:09:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.862953354s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.785708389s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:09:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.785725734s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.813355706s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:09:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.813379291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [..........................X............X..........X.........] duration: 4.681719474s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:09:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 4.681744401s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.167300794s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:09:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 5.167324879s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [X..........................X............X..........X........] duration: 4.208382233s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:09:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 4.208405476s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.727901409s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:09:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.727926256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.463613444s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:08:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.463636728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.776106255s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:08:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.776144536s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.X..........................X............X..........X.......] duration: 4.13954236s trace: https://jaeger.dev.storj.io/trace/c70cedfa074a735 FAIL (2026-05-26 04:08:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5DR3AA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787482465 (25h58m29.570822564s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787491221 (25h58m29.57083132s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787929247 (25h58m29.571269346s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.787939006 (25h58m29.571279105s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788227547 (25h58m29.571567656s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788485719 (25h58m29.571825818s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788551093 (25h58m29.571891192s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788807573 (25h58m29.572147682s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.788839965 (25h58m29.572180064s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789161868 (25h58m29.572501968s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789306404 (25h58m29.572646504s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789440411 (25h58m29.57278051s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789696449 (25h58m29.573036548s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.789750442 (25h58m29.573090541s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:43.790047749 (25h58m29.573387848s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Uploading piece... May 26 04:02:44.855320549 (25h58m30.638660649s) [upload=24845]: Next returned nil segment May 26 04:02:44.855334285 (25h58m30.638674384s) [upload=24845]: Waiting for error group managing segments... May 26 04:02:45.136813366 (25h58m30.920153465s) [noise=true , address=212.102.52.216:4398 , piece_id=UE6H5RRIE3BPNS2X6XLX77UPWJW37VGNDTIRQISHQQJBZCBZS3LA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.136955096 (25h58m30.920295195s) [noise=true , address=212.102.52.213:4326 , piece_id=F6PLYBPVD7B3GXJLDRXL2JFX5YATOYHLPKLUZXTQUXWQBV2UH3UA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142017395 (25h58m30.925357494s) [noise=true , address=212.102.52.216:4406 , piece_id=SMSYFCNW72YACSYKPTRBFXMKUWF67BBS74A3RRMG65TTIGOQNNEQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.142710257 (25h58m30.926050346s) [noise=true , address=212.102.52.214:4364 , piece_id=RU3TUJ3SYIXULCBETMQNI7L2ZI5ZNOIEZGTAZS35EQZNEUH3XTMQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.143856514 (25h58m30.927196613s) [noise=true , address=212.102.52.217:4436 , piece_id=SMZ5MRMUOUJ5JUPE2BTDIQQQQHJHJW6N76IQQGCTKUOK4C7BVIWQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.147566854 (25h58m30.930906953s) [noise=true , address=212.102.52.217:4418 , piece_id=A3C25RTVHNTXJDSF7PEXPFHITYIKFMVJCIH6OQK5NJSUFDULDR5A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148110141 (25h58m30.93145024s) [noise=true , address=212.102.52.213:4321 , piece_id=ORUNY6WALJMAXADCPK4R7HMXNWFUN7JQEOSZRVTKKL7IWKABXPXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.148515774 (25h58m30.931855873s) [noise=true , address=212.102.52.216:4397 , piece_id=5KRS3KXQMDHDPCUEUWZA4WHYOBRYFZF2EIC7ZDU2BFDELFV6WQZQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150498498 (25h58m30.933838597s) [noise=true , address=212.102.52.214:4352 , piece_id=YCGUBQOTD2EBWJJN77AUGMB7FROKY3NS5LJX53H76LFGAIGOAWHA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.150838105 (25h58m30.934178204s) [noise=true , address=212.102.52.217:4417 , piece_id=2M5JPMIB5OXAXB2JEGJ5GV5SNAMFAXN3D4ICDXMX5L3WOU5DR3AA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.153937318 (25h58m30.937277417s) [noise=true , address=212.102.52.215:4372 , piece_id=GPM7VQADPDFEPNRDCW6ME24NAUR7AAQVIXBFLFXDKH25TFJ6TOCQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.155703559 (25h58m30.939043658s) [noise=true , address=212.102.52.215:4385 , piece_id=RNP74ZVVUOJO43PP4U6FCG77MZWRW4F54O3Z5Y5XCZOVIZ3GYKKA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.160568171 (25h58m30.94390827s) [noise=true , address=212.102.52.214:4366 , piece_id=KVRNKJBTYV7OS4A6KBSYAJLAFLMX7MS2ORB3FZKDUZJQR6NB32LQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.166869513 (25h58m30.950209612s) [noise=true , address=212.102.52.214:4357 , piece_id=BNJUWJCVB5QGUTPCM2ZBW4Y4QM54EUCJBFI7AA25WELKUZAEAMSQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.196317291 (25h58m30.97965739s) [noise=true , address=212.102.52.213:4329 , piece_id=ACKQHUZASRUPFPOSQLUU4SDE3HFIAECQWEMWJEPQQLZFPU55FO3A, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198032654 (25h58m30.981372753s) [noise=true , address=212.102.52.215:4381 , piece_id=N6AB2K2RP6ZYOUZ4B6HZNCXRNYSBLGOMVVVRY32NVFDZQL6D4PXA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: <nil> May 26 04:02:45.198052431 (25h58m30.98139253s) [seg_pos={0 0}, upload=24845]: Satisfied cohort requirements after 16 pieces May 26 04:02:45.198235120 (25h58m30.981575219s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y7fgkR8bVb2KyytzjpKenEJVq3vrWEmbrVL9aMzc1ZfFjkURS): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198241873 (25h58m30.981581972s) [noise=true , address=212.102.52.215:4384 , piece_id=25ZTRM25CFNWNQP72GAMUQDX5OQFA676HCKMGB6UL45UGFDSBAJQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198354898 (25h58m30.981694998s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198363104 (25h58m30.981703203s) [noise=true , address=212.102.52.217:4427 , piece_id=GBGGANJYEDTOS2DKSI5NOTEVIG6IZ25IYHG7SK42EMQFUGBK26RQ, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198430133 (25h58m30.981770232s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BsdeZy1odVCAgcjxU6KdveXhUUTJKviRrrHLNCUuVergW6zVk): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198436104 (25h58m30.981776203s) [noise=true , address=212.102.52.216:4408 , piece_id=ANEVUGHIQZ7F52TS6PCW7UIWNGDJDDW43O6UQ3NGU2R6L5UVUSNA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198539751 (25h58m30.98187985s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MGexEpM3Yefmhe4mYARRJHvoGH4hNfULwt2uRgFs9fs18D9Md): protocol: expected piece hash; context canceled; EOF May 26 04:02:45.198544520 (25h58m30.981884619s) [noise=true , address=212.102.52.213:4327 , piece_id=JT3FEE2MFRUM46Y7B2SFSOPW2V7FMKN7J3DXGGFIT7HBB3MJSAAA, seg_pos={0 0}, upload=24845]: Long tail cancellation due to optimal threshold May 26 04:02:45.198561443 (25h58m30.981901542s) [seg_pos={0 0}, upload=24845]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 04:02:45.198633259 (25h58m30.981973358s) [upload=24845]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 04:02:45.198637639 (25h58m30.981977738s) [upload=24845]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 04:02:50.550253682 (25h58m36.333593781s) [upload=24845]: Done uploading segments... May 26 04:02:50.550269892 (25h58m36.333609991s) [upload=24845]: Upload finished. err: <nil> Duration: 4.13955838s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.706111302s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.706133896s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.224689579s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:08:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.224711341s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.886665117s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 04:08:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.886719881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 15.831527425s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 04:08:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.831559426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.557997652s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 04:07:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.558026336s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.163707566s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 04:07:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 5.1637308s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)