Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X...XX.XXXX.X..................X..........X.XX.X....X....X.] duration: 50.220693646s trace: https://jaeger.dev.storj.io/trace/4cd8fa1e4f5e1dc8 FAIL (2025-12-26 15:46:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: 5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA) context deadline exceeded; piecestore: (Node ID: 12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, Piece ID: 5MOGOYCIZDUF3RBK7KBVNJDKS5JONLHBG6D2UVZUDL45MWN2P4TA) context deadline exceeded; piecestore: (Node ID: 12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, Piece ID: Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ) context deadline exceeded ======LOGS====== 629 (504h15m24.782960674s) [noise=true , address=145.79.227.52:4449 , piece_id=TZBWOOBMMSPSNEWKBLAAHDRSKLCBDKRZQC3KXCCRH7QCHQ7QJUYA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912000853 (504h15m24.783083918s) [noise=true , address=145.79.227.34:4923 , piece_id=PPAPN6DNQG5RBVMEW7PJWW37KULMSHFBZ24AGMMDQ7IPUWZGETAQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912414501 (504h15m24.783497546s) [noise=true , address=77.90.31.17:4799 , piece_id=SQISFPF6L5V6JW6BP2COCZF7Z4X3BPUE2D3SWTHH3EKNJM5WZFXQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912565018 (504h15m24.783648063s) [noise=true , address=145.79.227.16:4914 , piece_id=EBYZPSEJGMOPUKXTDJ3PWIYBPWVVWKQB7TQ2EJM3CVOIZH66GSQA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912793312 (504h15m24.783876357s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.913155432 (504h15m24.784238477s) [noise=true , address=77.90.31.21:4895 , piece_id=5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.913447237 (504h15m24.784530272s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.914073700 (504h15m24.785156755s) [noise=true , address=145.79.227.40:4055 , piece_id=5LHHIMTDTOAIMBJBGQJHTVYK4XHV2CDIFKCQJ464LGZF56LVHGTQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.914654306 (504h15m24.785737351s) [noise=true , address=64.49.13.27:4961 , piece_id=SWD24N7WNLCAAWI7PJYLSLH6MOXXO44RSJNMNFHWUJ3NFPIUP6YA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.915261752 (504h15m24.786344798s) [noise=true , address=45.91.116.10:4092 , piece_id=LFKFNJRC7VZM3YKJRESLYNM746DG2QQDBR2CMKOJWQKGWWVXE62Q, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.915584346 (504h15m24.786667392s) [noise=true , address=77.90.31.15:4708 , piece_id=4TZTHMBJ7JLAKD4PTWBG5OBS2SHM7ARDW4MX2WGYSB5BOOAVSDPA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916034153 (504h15m24.787117188s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916473640 (504h15m24.787556675s) [noise=true , address=77.90.31.18:4836 , piece_id=6K7KXGECA6OM6UPZ5QYK246WOHQTB3M4NSXNHJX3BYSYLIZNVPSQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916980105 (504h15m24.78806315s) [noise=true , address=77.90.31.16:4751 , piece_id=Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.917377612 (504h15m24.788460657s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:26.223735515 (504h15m26.09481856s) [upload=442494]: Next returned nil segment Dec 26 15:44:26.223748189 (504h15m26.094831224s) [upload=442494]: Waiting for error group managing segments... Dec 26 15:44:26.592105696 (504h15m26.463188742s) [noise=true , address=64.49.13.137:4499 , piece_id=T5CKU2H2XUCA4OWMOM7RHK7G32EEJTCHOENTBDXZEH56A7QHCSCA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.791746046 (504h15m26.662829091s) [noise=true , address=145.79.227.16:4914 , piece_id=EBYZPSEJGMOPUKXTDJ3PWIYBPWVVWKQB7TQ2EJM3CVOIZH66GSQA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.866532121 (504h15m26.737615166s) [noise=true , address=64.49.13.27:4961 , piece_id=SWD24N7WNLCAAWI7PJYLSLH6MOXXO44RSJNMNFHWUJ3NFPIUP6YA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.945299659 (504h15m26.816382694s) [noise=true , address=145.79.227.52:4449 , piece_id=TZBWOOBMMSPSNEWKBLAAHDRSKLCBDKRZQC3KXCCRH7QCHQ7QJUYA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.097735153 (504h15m26.968818188s) [noise=true , address=77.90.31.13:4625 , piece_id=KH3MOO4QFLK237MTUUG7DJZOJT6P3UQFHR4PKVL4SXHU5N32BDVA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.220855005 (504h15m27.09193804s) [noise=true , address=77.90.31.21:4895 , piece_id=5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.339482502 (504h15m27.210565547s) [noise=true , address=77.90.31.17:4799 , piece_id=SQISFPF6L5V6JW6BP2COCZF7Z4X3BPUE2D3SWTHH3EKNJM5WZFXQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.637331558 (504h15m27.508414603s) [noise=true , address=77.90.31.12:4621 , piece_id=5MOGOYCIZDUF3RBK7KBVNJDKS5JONLHBG6D2UVZUDL45MWN2P4TA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.708229921 (504h15m27.579312966s) [noise=true , address=145.79.227.40:4055 , piece_id=5LHHIMTDTOAIMBJBGQJHTVYK4XHV2CDIFKCQJ464LGZF56LVHGTQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.739740644 (504h15m27.610823689s) [noise=true , address=145.79.227.34:4923 , piece_id=PPAPN6DNQG5RBVMEW7PJWW37KULMSHFBZ24AGMMDQ7IPUWZGETAQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.746887031 (504h15m27.617970076s) [noise=true , address=77.90.31.15:4708 , piece_id=4TZTHMBJ7JLAKD4PTWBG5OBS2SHM7ARDW4MX2WGYSB5BOOAVSDPA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.979849332 (504h15m27.850932367s) [noise=true , address=77.90.31.18:4836 , piece_id=6K7KXGECA6OM6UPZ5QYK246WOHQTB3M4NSXNHJX3BYSYLIZNVPSQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.390591527 (504h15m28.261674562s) [noise=true , address=77.90.31.16:4751 , piece_id=Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.569880514 (504h15m28.440963559s) [noise=true , address=45.91.116.10:4092 , piece_id=LFKFNJRC7VZM3YKJRESLYNM746DG2QQDBR2CMKOJWQKGWWVXE62Q, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.569901214 (504h15m28.440984249s) [seg_pos={0 0}, upload=442494]: Segment reached optimal threshold of 14 pieces. Dec 26 15:44:28.570240661 (504h15m28.441323706s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 26 15:44:28.570256761 (504h15m28.441339796s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570262151 (504h15m28.441345186s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection Dec 26 15:44:28.570271910 (504h15m28.441354945s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570537195 (504h15m28.44162023s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: rpc: dial tcp 45.91.116.4:4948: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:44:28.570544318 (504h15m28.441627353s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570740913 (504h15m28.441823958s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: read tcp 188.166.225.170:59608->45.91.116.6:4999: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:44:28.570753126 (504h15m28.441836161s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570780768 (504h15m28.441863814s) [seg_pos={0 0}, upload=442494]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:44:28.570883895 (504h15m28.44196693s) [upload=442494]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:44:28.570907761 (504h15m28.441990816s) [upload=442494]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:44:28.878863402 (504h15m28.749946477s) [upload=442494]: Done uploading segments... Dec 26 15:44:28.878894812 (504h15m28.749977887s) [upload=442494]: Upload finished. err: <nil> Duration: 50.220715055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.....X.X............XXX....XX...X......X..X..X.X......XX.] duration: 7.612363828s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:45:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 7.612389348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X.....X.X............XXX....XX...X......X..X..X.X......XX] duration: 30.004015833s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:45:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 30.004027545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X...XX.XXXX.X..................X..........X.XX.X....X....X] duration: 1m0.003275518s trace: https://jaeger.dev.storj.io/trace/4cd8fa1e4f5e1dc8 FAIL (2025-12-26 15:45:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: 5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA) context deadline exceeded; piecestore: (Node ID: 12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, Piece ID: 5MOGOYCIZDUF3RBK7KBVNJDKS5JONLHBG6D2UVZUDL45MWN2P4TA) context deadline exceeded; piecestore: (Node ID: 12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, Piece ID: Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ) context deadline exceeded ======LOGS====== 629 (504h15m24.782960674s) [noise=true , address=145.79.227.52:4449 , piece_id=TZBWOOBMMSPSNEWKBLAAHDRSKLCBDKRZQC3KXCCRH7QCHQ7QJUYA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912000853 (504h15m24.783083918s) [noise=true , address=145.79.227.34:4923 , piece_id=PPAPN6DNQG5RBVMEW7PJWW37KULMSHFBZ24AGMMDQ7IPUWZGETAQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912414501 (504h15m24.783497546s) [noise=true , address=77.90.31.17:4799 , piece_id=SQISFPF6L5V6JW6BP2COCZF7Z4X3BPUE2D3SWTHH3EKNJM5WZFXQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912565018 (504h15m24.783648063s) [noise=true , address=145.79.227.16:4914 , piece_id=EBYZPSEJGMOPUKXTDJ3PWIYBPWVVWKQB7TQ2EJM3CVOIZH66GSQA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912793312 (504h15m24.783876357s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.913155432 (504h15m24.784238477s) [noise=true , address=77.90.31.21:4895 , piece_id=5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.913447237 (504h15m24.784530272s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.914073700 (504h15m24.785156755s) [noise=true , address=145.79.227.40:4055 , piece_id=5LHHIMTDTOAIMBJBGQJHTVYK4XHV2CDIFKCQJ464LGZF56LVHGTQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.914654306 (504h15m24.785737351s) [noise=true , address=64.49.13.27:4961 , piece_id=SWD24N7WNLCAAWI7PJYLSLH6MOXXO44RSJNMNFHWUJ3NFPIUP6YA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.915261752 (504h15m24.786344798s) [noise=true , address=45.91.116.10:4092 , piece_id=LFKFNJRC7VZM3YKJRESLYNM746DG2QQDBR2CMKOJWQKGWWVXE62Q, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.915584346 (504h15m24.786667392s) [noise=true , address=77.90.31.15:4708 , piece_id=4TZTHMBJ7JLAKD4PTWBG5OBS2SHM7ARDW4MX2WGYSB5BOOAVSDPA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916034153 (504h15m24.787117188s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916473640 (504h15m24.787556675s) [noise=true , address=77.90.31.18:4836 , piece_id=6K7KXGECA6OM6UPZ5QYK246WOHQTB3M4NSXNHJX3BYSYLIZNVPSQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916980105 (504h15m24.78806315s) [noise=true , address=77.90.31.16:4751 , piece_id=Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.917377612 (504h15m24.788460657s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:26.223735515 (504h15m26.09481856s) [upload=442494]: Next returned nil segment Dec 26 15:44:26.223748189 (504h15m26.094831224s) [upload=442494]: Waiting for error group managing segments... Dec 26 15:44:26.592105696 (504h15m26.463188742s) [noise=true , address=64.49.13.137:4499 , piece_id=T5CKU2H2XUCA4OWMOM7RHK7G32EEJTCHOENTBDXZEH56A7QHCSCA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.791746046 (504h15m26.662829091s) [noise=true , address=145.79.227.16:4914 , piece_id=EBYZPSEJGMOPUKXTDJ3PWIYBPWVVWKQB7TQ2EJM3CVOIZH66GSQA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.866532121 (504h15m26.737615166s) [noise=true , address=64.49.13.27:4961 , piece_id=SWD24N7WNLCAAWI7PJYLSLH6MOXXO44RSJNMNFHWUJ3NFPIUP6YA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.945299659 (504h15m26.816382694s) [noise=true , address=145.79.227.52:4449 , piece_id=TZBWOOBMMSPSNEWKBLAAHDRSKLCBDKRZQC3KXCCRH7QCHQ7QJUYA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.097735153 (504h15m26.968818188s) [noise=true , address=77.90.31.13:4625 , piece_id=KH3MOO4QFLK237MTUUG7DJZOJT6P3UQFHR4PKVL4SXHU5N32BDVA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.220855005 (504h15m27.09193804s) [noise=true , address=77.90.31.21:4895 , piece_id=5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.339482502 (504h15m27.210565547s) [noise=true , address=77.90.31.17:4799 , piece_id=SQISFPF6L5V6JW6BP2COCZF7Z4X3BPUE2D3SWTHH3EKNJM5WZFXQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.637331558 (504h15m27.508414603s) [noise=true , address=77.90.31.12:4621 , piece_id=5MOGOYCIZDUF3RBK7KBVNJDKS5JONLHBG6D2UVZUDL45MWN2P4TA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.708229921 (504h15m27.579312966s) [noise=true , address=145.79.227.40:4055 , piece_id=5LHHIMTDTOAIMBJBGQJHTVYK4XHV2CDIFKCQJ464LGZF56LVHGTQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.739740644 (504h15m27.610823689s) [noise=true , address=145.79.227.34:4923 , piece_id=PPAPN6DNQG5RBVMEW7PJWW37KULMSHFBZ24AGMMDQ7IPUWZGETAQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.746887031 (504h15m27.617970076s) [noise=true , address=77.90.31.15:4708 , piece_id=4TZTHMBJ7JLAKD4PTWBG5OBS2SHM7ARDW4MX2WGYSB5BOOAVSDPA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.979849332 (504h15m27.850932367s) [noise=true , address=77.90.31.18:4836 , piece_id=6K7KXGECA6OM6UPZ5QYK246WOHQTB3M4NSXNHJX3BYSYLIZNVPSQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.390591527 (504h15m28.261674562s) [noise=true , address=77.90.31.16:4751 , piece_id=Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.569880514 (504h15m28.440963559s) [noise=true , address=45.91.116.10:4092 , piece_id=LFKFNJRC7VZM3YKJRESLYNM746DG2QQDBR2CMKOJWQKGWWVXE62Q, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.569901214 (504h15m28.440984249s) [seg_pos={0 0}, upload=442494]: Segment reached optimal threshold of 14 pieces. Dec 26 15:44:28.570240661 (504h15m28.441323706s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 26 15:44:28.570256761 (504h15m28.441339796s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570262151 (504h15m28.441345186s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection Dec 26 15:44:28.570271910 (504h15m28.441354945s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570537195 (504h15m28.44162023s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: rpc: dial tcp 45.91.116.4:4948: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:44:28.570544318 (504h15m28.441627353s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570740913 (504h15m28.441823958s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: read tcp 188.166.225.170:59608->45.91.116.6:4999: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:44:28.570753126 (504h15m28.441836161s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570780768 (504h15m28.441863814s) [seg_pos={0 0}, upload=442494]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:44:28.570883895 (504h15m28.44196693s) [upload=442494]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:44:28.570907761 (504h15m28.441990816s) [upload=442494]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:44:28.878863402 (504h15m28.749946477s) [upload=442494]: Done uploading segments... Dec 26 15:44:28.878894812 (504h15m28.749977887s) [upload=442494]: Upload finished. err: <nil> Duration: 1m0.003291848s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X...] duration: 4.933983715s trace: https://jaeger.dev.storj.io/trace/3872e6d9920cd189 FAIL (2025-12-26 15:45:08 UTC) orbiter warning: context deadline exceeded Duration: 4.934076723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.....X.X............XXX....XX...X......X..X..X.X......X] duration: 30.003203896s trace: https://jaeger.dev.storj.io/trace/4812822751f78e6c FAIL (2025-12-26 15:44:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 3667XNMBF55ZSGWI4VIBBZBPC4VFYS3XW4NPT6N5ZSFRSBASLXQA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: LTBYEM5KP6WSWKBGAQKC6T7Y2AMT6WPD7O2PUIV6UWZOHT7CAH7Q) context deadline exceeded ======LOGS====== 26 15:44:29.736880526 (504h15m29.607963572s) [upload=442495]: Starting upload Dec 26 15:44:29.736970237 (504h15m29.608053282s) [upload=442495]: Done starting upload Dec 26 15:44:29.737195596 (504h15m29.608278641s) [upload=442495]: Uploading segments... Dec 26 15:44:29.737264868 (504h15m29.608347933s) [upload=442495]: Got next segment. Inline: false Dec 26 15:44:29.737303081 (504h15m29.608386126s) [upload=442495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:29.737309703 (504h15m29.608392748s) [upload=442495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:44:29.983933468 (504h15m29.855016503s) [seg_pos={0 0}, upload=442495]: Begin upload segment... Dec 26 15:44:29.983979525 (504h15m29.85506256s) [seg_pos={0 0}, upload=442495]: Done begin upload segment. Dec 26 15:44:29.984017978 (504h15m29.855101013s) [noise=true , address=77.90.31.21:4880 , piece_id=4FJR4VRRXC4OWXCRNY2C6RARU3LJRFSYRWBNH2KLXDQULWDD246A, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.985315780 (504h15m29.856398815s) [noise=true , address=145.79.227.3:4149 , piece_id=DFUEZB67CXCZXXM27XU7IHOTR3UDJC76WY3WI2WO2UMYO3SAGKAA, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.986073824 (504h15m29.857156859s) [noise=true , address=145.79.227.46:4603 , piece_id=FUKO5VESS6T2XU5ENHUC4JLN2ZFQGRP72TAIKELDBAX3L3HIXLBQ, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.986478374 (504h15m29.857561419s) [noise=true , address=145.79.227.44:4194 , piece_id=HDQSR6ZMKRN3LXXSRGRBGMHVPFJA3OWNEZBQBKTQ2IYJHZMTZWCA, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.986777834 (504h15m29.857860869s) [noise=true , address=145.79.227.5:4222 , piece_id=KTDGWVAOAXLPFMEIYFW6C7F2GUDK7KGQJSMQB667T7OBPM4OAGTQ, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.987136036 (504h15m29.858219081s) [noise=true , address=45.91.116.8:4047 , piece_id=MZP5RDDTMIEC3O3U3SO62MDAJD6LNFE6BQRTKVISCRBIZH6MATBQ, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.987880533 (504h15m29.858963568s) [noise=true , address=145.79.227.50:4374 , piece_id=FKBNENC6R2QVFQMLROFO6YD4I4MK4PRUMHH2HHEMVBEX47DNKMFA, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.988368573 (504h15m29.859451608s) [noise=true , address=145.79.227.17:4971 , piece_id=X2GPRFPPMV27VXRBAF4II67Y7VSDZQ6ATQK5ZYM5346XXHIR4ARQ, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.988923209 (504h15m29.860006254s) [noise=true , address=145.79.227.48:4303 , piece_id=VV2FNK2UDD67EDV6EW36YG4R5BZGWS6IYK2B6F6K7G6KMJSFX6GQ, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.989534052 (504h15m29.860617118s) [noise=true , address=45.91.116.4:4940 , piece_id=2DYC4YKALQSFB3SLJLZKUNMITI3ESQAK44SOSBZR4YG6KZUU7NAA, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.990185513 (504h15m29.861268558s) [noise=true , address=145.79.227.49:4339 , piece_id=SIITQIKKG6LE6WNQBRJZLK6TEF2X5U446KMC7BI6MREMZAGN6BPA, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.990887009 (504h15m29.861970044s) [noise=true , address=45.91.116.5:4298 , piece_id=PNBNKSM5PNXKRULZAYYKAR47UFCEBMIIEJBTYYZZAI4GT2RFIN3Q, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.991532818 (504h15m29.862615853s) [noise=true , address=45.91.116.13:4156 , piece_id=3667XNMBF55ZSGWI4VIBBZBPC4VFYS3XW4NPT6N5ZSFRSBASLXQA, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.992554644 (504h15m29.863637689s) [noise=true , address=145.79.227.35:4483 , piece_id=7CBMVRKP4NRWZKMRZOM6GS3Y2BLPKDCMO2VOSPR2YS6SWF4UZX7A, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.992932965 (504h15m29.86401601s) [noise=true , address=145.79.227.43:4159 , piece_id=DNG7U4E23IC6CCNOCXCJLHI7LFDJ5ANZUDIIXEJVEDU2FESPGKWQ, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:29.995047020 (504h15m29.866130055s) [noise=true , address=45.91.116.10:4084 , piece_id=LTBYEM5KP6WSWKBGAQKC6T7Y2AMT6WPD7O2PUIV6UWZOHT7CAH7Q, seg_pos={0 0}, upload=442495]: Uploading piece... Dec 26 15:44:31.705281044 (504h15m31.576364089s) [upload=442495]: Next returned nil segment Dec 26 15:44:31.705296593 (504h15m31.576379638s) [upload=442495]: Waiting for error group managing segments... Dec 26 15:44:31.994705459 (504h15m31.865788494s) [noise=true , address=145.79.227.49:4339 , piece_id=SIITQIKKG6LE6WNQBRJZLK6TEF2X5U446KMC7BI6MREMZAGN6BPA, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:31.995195792 (504h15m31.866278838s) [noise=true , address=145.79.227.46:4603 , piece_id=FUKO5VESS6T2XU5ENHUC4JLN2ZFQGRP72TAIKELDBAX3L3HIXLBQ, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:31.998161039 (504h15m31.869244084s) [noise=true , address=145.79.227.3:4149 , piece_id=DFUEZB67CXCZXXM27XU7IHOTR3UDJC76WY3WI2WO2UMYO3SAGKAA, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:32.015593113 (504h15m31.886676148s) [noise=true , address=145.79.227.17:4971 , piece_id=X2GPRFPPMV27VXRBAF4II67Y7VSDZQ6ATQK5ZYM5346XXHIR4ARQ, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:32.020205114 (504h15m31.891288149s) [noise=true , address=145.79.227.35:4483 , piece_id=7CBMVRKP4NRWZKMRZOM6GS3Y2BLPKDCMO2VOSPR2YS6SWF4UZX7A, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:32.022352894 (504h15m31.893435939s) [noise=true , address=145.79.227.5:4222 , piece_id=KTDGWVAOAXLPFMEIYFW6C7F2GUDK7KGQJSMQB667T7OBPM4OAGTQ, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:32.030103763 (504h15m31.901186799s) [noise=true , address=145.79.227.50:4374 , piece_id=FKBNENC6R2QVFQMLROFO6YD4I4MK4PRUMHH2HHEMVBEX47DNKMFA, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:32.030540525 (504h15m31.90162356s) [noise=true , address=145.79.227.48:4303 , piece_id=VV2FNK2UDD67EDV6EW36YG4R5BZGWS6IYK2B6F6K7G6KMJSFX6GQ, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:32.188763970 (504h15m32.059847005s) [noise=true , address=77.90.31.21:4880 , piece_id=4FJR4VRRXC4OWXCRNY2C6RARU3LJRFSYRWBNH2KLXDQULWDD246A, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:32.349828575 (504h15m32.22091161s) [noise=true , address=45.91.116.13:4156 , piece_id=3667XNMBF55ZSGWI4VIBBZBPC4VFYS3XW4NPT6N5ZSFRSBASLXQA, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:32.356345364 (504h15m32.227428399s) [noise=true , address=45.91.116.4:4940 , piece_id=2DYC4YKALQSFB3SLJLZKUNMITI3ESQAK44SOSBZR4YG6KZUU7NAA, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:33.010263468 (504h15m32.881346513s) [noise=true , address=45.91.116.10:4084 , piece_id=LTBYEM5KP6WSWKBGAQKC6T7Y2AMT6WPD7O2PUIV6UWZOHT7CAH7Q, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:33.196189164 (504h15m33.067272199s) [noise=true , address=45.91.116.5:4298 , piece_id=PNBNKSM5PNXKRULZAYYKAR47UFCEBMIIEJBTYYZZAI4GT2RFIN3Q, seg_pos={0 0}, upload=442495]: Done uploading piece. err: <nil> Dec 26 15:44:33.196215192 (504h15m33.067298227s) [seg_pos={0 0}, upload=442495]: Segment reached optimal threshold of 13 pieces. Dec 26 15:44:33.196377722 (504h15m33.067460757s) [noise=true , address=145.79.227.44:4194 , piece_id=HDQSR6ZMKRN3LXXSRGRBGMHVPFJA3OWNEZBQBKTQ2IYJHZMTZWCA, seg_pos={0 0}, upload=442495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 15:44:33.196383152 (504h15m33.067466197s) [noise=true , address=145.79.227.44:4194 , piece_id=HDQSR6ZMKRN3LXXSRGRBGMHVPFJA3OWNEZBQBKTQ2IYJHZMTZWCA, seg_pos={0 0}, upload=442495]: Long tail cancellation due to optimal threshold Dec 26 15:44:33.196452765 (504h15m33.06753583s) [noise=true , address=45.91.116.8:4047 , piece_id=MZP5RDDTMIEC3O3U3SO62MDAJD6LNFE6BQRTKVISCRBIZH6MATBQ, seg_pos={0 0}, upload=442495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:54948->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:54948->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54948->45.91.116.8:4047: use of closed network connection Dec 26 15:44:33.196463787 (504h15m33.067546832s) [noise=true , address=45.91.116.8:4047 , piece_id=MZP5RDDTMIEC3O3U3SO62MDAJD6LNFE6BQRTKVISCRBIZH6MATBQ, seg_pos={0 0}, upload=442495]: Long tail cancellation due to optimal threshold Dec 26 15:44:33.196494264 (504h15m33.067577299s) [noise=true , address=145.79.227.43:4159 , piece_id=DNG7U4E23IC6CCNOCXCJLHI7LFDJ5ANZUDIIXEJVEDU2FESPGKWQ, seg_pos={0 0}, upload=442495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 15:44:33.196499725 (504h15m33.06758276s) [noise=true , address=145.79.227.43:4159 , piece_id=DNG7U4E23IC6CCNOCXCJLHI7LFDJ5ANZUDIIXEJVEDU2FESPGKWQ, seg_pos={0 0}, upload=442495]: Long tail cancellation due to optimal threshold Dec 26 15:44:33.196513601 (504h15m33.067596636s) [seg_pos={0 0}, upload=442495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:44:33.196579016 (504h15m33.067662061s) [upload=442495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:44:33.196583384 (504h15m33.067666429s) [upload=442495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:44:33.496423281 (504h15m33.367506336s) [upload=442495]: Done uploading segments... Dec 26 15:44:33.496446776 (504h15m33.367529811s) [upload=442495]: Upload finished. err: <nil> Duration: 30.003224796s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X..] duration: 4.568370975s trace: https://jaeger.dev.storj.io/trace/3872e6d9920cd189 FAIL (2025-12-26 15:44:40 UTC) orbiter warning: context deadline exceeded Duration: 4.56839977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X...XX.XXXX.X..................X..........X.XX.X....X....] duration: 48.705077397s trace: https://jaeger.dev.storj.io/trace/dfc94eb2ec733dd FAIL (2025-12-26 15:44:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.079395532 (504h11m16.950478577s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.081581986 (504h11m16.952665031s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082159214 (504h11m16.953242259s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082917899 (504h11m16.954000944s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083135794 (504h11m16.954218839s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083289507 (504h11m16.954372542s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083912283 (504h11m16.954995318s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.084313877 (504h11m16.955396922s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085026795 (504h11m16.95610984s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085578926 (504h11m16.956661971s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085601299 (504h11m16.956684344s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086213805 (504h11m16.95729685s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086714428 (504h11m16.957797473s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.087503671 (504h11m16.958586727s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:18.448801149 (504h11m18.319884194s) [upload=442433]: Next returned nil segment Dec 26 15:40:18.448816528 (504h11m18.319899563s) [upload=442433]: Waiting for error group managing segments... Dec 26 15:40:18.795258594 (504h11m18.666341629s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.980610968 (504h11m18.851694013s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.992478436 (504h11m18.863561481s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.168471206 (504h11m19.039554242s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.198368747 (504h11m19.069451792s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.230854365 (504h11m19.1019374s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.291407819 (504h11m19.162490875s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.305940752 (504h11m19.177023787s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.412658281 (504h11m19.283741326s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.416084726 (504h11m19.287167771s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.863940842 (504h11m19.735023887s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.872450075 (504h11m19.74353312s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.918830151 (504h11m19.789913196s) [noise=true , address=145.79.227.34:4924 , piece_id=TTVUB2HO2U4LYY6RCT7FOHX4KQ3BBJRJF6UXXAZIPILINMMGUI3A, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952518840 (504h11m19.823601875s) [noise=true , address=145.79.227.30:4779 , piece_id=HM64K7CTAIV6HYTM64KTSXDFP5GZZVHCN3D6WY5ZH6DQ7XZDHHVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952540902 (504h11m19.823623947s) [seg_pos={0 0}, upload=442433]: Segment reached optimal threshold of 14 pieces. Dec 26 15:40:19.952728510 (504h11m19.823811545s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection Dec 26 15:40:19.952735372 (504h11m19.823818407s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952923771 (504h11m19.824006806s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection Dec 26 15:40:19.952958006 (504h11m19.824041051s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952976081 (504h11m19.824059126s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.952983415 (504h11m19.82406646s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953228752 (504h11m19.824311787s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T4C5mqpUTmi8sZq38KuFakx6Xh6FTVBn5jCFqFHr5GAyBBDaY): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.953235955 (504h11m19.824319s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953257617 (504h11m19.824340662s) [seg_pos={0 0}, upload=442433]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:40:19.953401942 (504h11m19.824484977s) [upload=442433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:40:19.953408223 (504h11m19.824491258s) [upload=442433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:40:20.252364659 (504h11m20.123447714s) [upload=442433]: Done uploading segments... Dec 26 15:40:20.252393865 (504h11m20.12347691s) [upload=442433]: Upload finished. err: <nil> Duration: 48.70509462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X.] duration: 4.37337569s trace: https://jaeger.dev.storj.io/trace/3872e6d9920cd189 FAIL (2025-12-26 15:43:58 UTC) orbiter warning: context deadline exceeded Duration: 4.373506511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.....X.X............XXX....XX...X......X..X..X.X......] duration: 5.89740617s trace: https://jaeger.dev.storj.io/trace/6fca4869d642b9be FAIL (2025-12-26 15:43:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ) context deadline exceeded ======LOGS====== Dec 26 15:38:11.884127603 (504h9m11.755210648s) [upload=442407]: Starting upload Dec 26 15:38:11.884220320 (504h9m11.755303375s) [upload=442407]: Done starting upload Dec 26 15:38:11.884476737 (504h9m11.755559773s) [upload=442407]: Uploading segments... Dec 26 15:38:11.884552192 (504h9m11.755635237s) [upload=442407]: Got next segment. Inline: false Dec 26 15:38:11.884592018 (504h9m11.755675063s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:38:11.884598049 (504h9m11.755681094s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:38:12.134094724 (504h9m12.005177759s) [seg_pos={0 0}, upload=442407]: Begin upload segment... Dec 26 15:38:12.134151552 (504h9m12.005234597s) [seg_pos={0 0}, upload=442407]: Done begin upload segment. Dec 26 15:38:12.134183844 (504h9m12.005266879s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134257975 (504h9m12.00534102s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134939473 (504h9m12.006022518s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135488959 (504h9m12.006572004s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135931852 (504h9m12.007014897s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136399252 (504h9m12.007482298s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136917600 (504h9m12.008000645s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.138384252 (504h9m12.009467297s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139126175 (504h9m12.01020921s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139807833 (504h9m12.010890868s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141360208 (504h9m12.012443254s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141717439 (504h9m12.012800485s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.143312085 (504h9m12.01439512s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.144758760 (504h9m12.015841795s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.146963618 (504h9m12.018046653s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.149176833 (504h9m12.020259878s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:13.860029024 (504h9m13.731112069s) [upload=442407]: Next returned nil segment Dec 26 15:38:13.860051176 (504h9m13.731134221s) [upload=442407]: Waiting for error group managing segments... Dec 26 15:38:14.140188417 (504h9m14.011271452s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.149246274 (504h9m14.020329319s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.150448424 (504h9m14.021531499s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.155601316 (504h9m14.026684361s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.157887398 (504h9m14.028970433s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.171654053 (504h9m14.042737098s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174269793 (504h9m14.045352828s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174722906 (504h9m14.045805941s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.399247390 (504h9m14.270330425s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.418217913 (504h9m14.289300948s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.524304791 (504h9m14.395387836s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.585548651 (504h9m14.456631696s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615716845 (504h9m14.48679988s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615737164 (504h9m14.4868202s) [seg_pos={0 0}, upload=442407]: Segment reached optimal threshold of 13 pieces. Dec 26 15:38:14.615911667 (504h9m14.486994702s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection Dec 26 15:38:14.615917758 (504h9m14.487000793s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616222659 (504h9m14.487305694s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616229632 (504h9m14.487312677s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616336716 (504h9m14.487419751s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616342036 (504h9m14.487425081s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616356093 (504h9m14.487439128s) [seg_pos={0 0}, upload=442407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:38:14.616417860 (504h9m14.487500895s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:38:14.616422590 (504h9m14.487505625s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:38:14.917908311 (504h9m14.788991356s) [upload=442407]: Done uploading segments... Dec 26 15:38:14.917924912 (504h9m14.789007957s) [upload=442407]: Upload finished. err: <nil> Duration: 5.897499347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.X...XX.XXXX.X..................X..........X.XX.X....X...] duration: 38.166526157s trace: https://jaeger.dev.storj.io/trace/dfc94eb2ec733dd FAIL (2025-12-26 15:43:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.079395532 (504h11m16.950478577s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.081581986 (504h11m16.952665031s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082159214 (504h11m16.953242259s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082917899 (504h11m16.954000944s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083135794 (504h11m16.954218839s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083289507 (504h11m16.954372542s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083912283 (504h11m16.954995318s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.084313877 (504h11m16.955396922s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085026795 (504h11m16.95610984s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085578926 (504h11m16.956661971s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085601299 (504h11m16.956684344s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086213805 (504h11m16.95729685s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086714428 (504h11m16.957797473s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.087503671 (504h11m16.958586727s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:18.448801149 (504h11m18.319884194s) [upload=442433]: Next returned nil segment Dec 26 15:40:18.448816528 (504h11m18.319899563s) [upload=442433]: Waiting for error group managing segments... Dec 26 15:40:18.795258594 (504h11m18.666341629s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.980610968 (504h11m18.851694013s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.992478436 (504h11m18.863561481s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.168471206 (504h11m19.039554242s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.198368747 (504h11m19.069451792s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.230854365 (504h11m19.1019374s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.291407819 (504h11m19.162490875s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.305940752 (504h11m19.177023787s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.412658281 (504h11m19.283741326s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.416084726 (504h11m19.287167771s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.863940842 (504h11m19.735023887s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.872450075 (504h11m19.74353312s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.918830151 (504h11m19.789913196s) [noise=true , address=145.79.227.34:4924 , piece_id=TTVUB2HO2U4LYY6RCT7FOHX4KQ3BBJRJF6UXXAZIPILINMMGUI3A, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952518840 (504h11m19.823601875s) [noise=true , address=145.79.227.30:4779 , piece_id=HM64K7CTAIV6HYTM64KTSXDFP5GZZVHCN3D6WY5ZH6DQ7XZDHHVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952540902 (504h11m19.823623947s) [seg_pos={0 0}, upload=442433]: Segment reached optimal threshold of 14 pieces. Dec 26 15:40:19.952728510 (504h11m19.823811545s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection Dec 26 15:40:19.952735372 (504h11m19.823818407s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952923771 (504h11m19.824006806s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection Dec 26 15:40:19.952958006 (504h11m19.824041051s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952976081 (504h11m19.824059126s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.952983415 (504h11m19.82406646s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953228752 (504h11m19.824311787s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T4C5mqpUTmi8sZq38KuFakx6Xh6FTVBn5jCFqFHr5GAyBBDaY): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.953235955 (504h11m19.824319s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953257617 (504h11m19.824340662s) [seg_pos={0 0}, upload=442433]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:40:19.953401942 (504h11m19.824484977s) [upload=442433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:40:19.953408223 (504h11m19.824491258s) [upload=442433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:40:20.252364659 (504h11m20.123447714s) [upload=442433]: Done uploading segments... Dec 26 15:40:20.252393865 (504h11m20.12347691s) [upload=442433]: Upload finished. err: <nil> Duration: 38.166543009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.....X.....X.X............XXX....XX...X......X..X..X.X.....] duration: 5.791691953s trace: https://jaeger.dev.storj.io/trace/6fca4869d642b9be FAIL (2025-12-26 15:43:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ) context deadline exceeded ======LOGS====== Dec 26 15:38:11.884127603 (504h9m11.755210648s) [upload=442407]: Starting upload Dec 26 15:38:11.884220320 (504h9m11.755303375s) [upload=442407]: Done starting upload Dec 26 15:38:11.884476737 (504h9m11.755559773s) [upload=442407]: Uploading segments... Dec 26 15:38:11.884552192 (504h9m11.755635237s) [upload=442407]: Got next segment. Inline: false Dec 26 15:38:11.884592018 (504h9m11.755675063s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:38:11.884598049 (504h9m11.755681094s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:38:12.134094724 (504h9m12.005177759s) [seg_pos={0 0}, upload=442407]: Begin upload segment... Dec 26 15:38:12.134151552 (504h9m12.005234597s) [seg_pos={0 0}, upload=442407]: Done begin upload segment. Dec 26 15:38:12.134183844 (504h9m12.005266879s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134257975 (504h9m12.00534102s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134939473 (504h9m12.006022518s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135488959 (504h9m12.006572004s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135931852 (504h9m12.007014897s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136399252 (504h9m12.007482298s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136917600 (504h9m12.008000645s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.138384252 (504h9m12.009467297s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139126175 (504h9m12.01020921s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139807833 (504h9m12.010890868s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141360208 (504h9m12.012443254s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141717439 (504h9m12.012800485s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.143312085 (504h9m12.01439512s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.144758760 (504h9m12.015841795s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.146963618 (504h9m12.018046653s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.149176833 (504h9m12.020259878s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:13.860029024 (504h9m13.731112069s) [upload=442407]: Next returned nil segment Dec 26 15:38:13.860051176 (504h9m13.731134221s) [upload=442407]: Waiting for error group managing segments... Dec 26 15:38:14.140188417 (504h9m14.011271452s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.149246274 (504h9m14.020329319s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.150448424 (504h9m14.021531499s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.155601316 (504h9m14.026684361s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.157887398 (504h9m14.028970433s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.171654053 (504h9m14.042737098s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174269793 (504h9m14.045352828s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174722906 (504h9m14.045805941s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.399247390 (504h9m14.270330425s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.418217913 (504h9m14.289300948s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.524304791 (504h9m14.395387836s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.585548651 (504h9m14.456631696s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615716845 (504h9m14.48679988s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615737164 (504h9m14.4868202s) [seg_pos={0 0}, upload=442407]: Segment reached optimal threshold of 13 pieces. Dec 26 15:38:14.615911667 (504h9m14.486994702s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection Dec 26 15:38:14.615917758 (504h9m14.487000793s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616222659 (504h9m14.487305694s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616229632 (504h9m14.487312677s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616336716 (504h9m14.487419751s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616342036 (504h9m14.487425081s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616356093 (504h9m14.487439128s) [seg_pos={0 0}, upload=442407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:38:14.616417860 (504h9m14.487500895s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:38:14.616422590 (504h9m14.487505625s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:38:14.917908311 (504h9m14.788991356s) [upload=442407]: Done uploading segments... Dec 26 15:38:14.917924912 (504h9m14.789007957s) [upload=442407]: Upload finished. err: <nil> Duration: 5.791714916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X.X...XX.XXXX.X..................X..........X.XX.X....X..] duration: 56.392001833s trace: https://jaeger.dev.storj.io/trace/dfc94eb2ec733dd FAIL (2025-12-26 15:42:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.079395532 (504h11m16.950478577s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.081581986 (504h11m16.952665031s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082159214 (504h11m16.953242259s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082917899 (504h11m16.954000944s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083135794 (504h11m16.954218839s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083289507 (504h11m16.954372542s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083912283 (504h11m16.954995318s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.084313877 (504h11m16.955396922s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085026795 (504h11m16.95610984s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085578926 (504h11m16.956661971s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085601299 (504h11m16.956684344s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086213805 (504h11m16.95729685s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086714428 (504h11m16.957797473s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.087503671 (504h11m16.958586727s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:18.448801149 (504h11m18.319884194s) [upload=442433]: Next returned nil segment Dec 26 15:40:18.448816528 (504h11m18.319899563s) [upload=442433]: Waiting for error group managing segments... Dec 26 15:40:18.795258594 (504h11m18.666341629s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.980610968 (504h11m18.851694013s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.992478436 (504h11m18.863561481s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.168471206 (504h11m19.039554242s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.198368747 (504h11m19.069451792s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.230854365 (504h11m19.1019374s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.291407819 (504h11m19.162490875s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.305940752 (504h11m19.177023787s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.412658281 (504h11m19.283741326s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.416084726 (504h11m19.287167771s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.863940842 (504h11m19.735023887s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.872450075 (504h11m19.74353312s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.918830151 (504h11m19.789913196s) [noise=true , address=145.79.227.34:4924 , piece_id=TTVUB2HO2U4LYY6RCT7FOHX4KQ3BBJRJF6UXXAZIPILINMMGUI3A, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952518840 (504h11m19.823601875s) [noise=true , address=145.79.227.30:4779 , piece_id=HM64K7CTAIV6HYTM64KTSXDFP5GZZVHCN3D6WY5ZH6DQ7XZDHHVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952540902 (504h11m19.823623947s) [seg_pos={0 0}, upload=442433]: Segment reached optimal threshold of 14 pieces. Dec 26 15:40:19.952728510 (504h11m19.823811545s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection Dec 26 15:40:19.952735372 (504h11m19.823818407s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952923771 (504h11m19.824006806s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection Dec 26 15:40:19.952958006 (504h11m19.824041051s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952976081 (504h11m19.824059126s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.952983415 (504h11m19.82406646s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953228752 (504h11m19.824311787s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T4C5mqpUTmi8sZq38KuFakx6Xh6FTVBn5jCFqFHr5GAyBBDaY): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.953235955 (504h11m19.824319s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953257617 (504h11m19.824340662s) [seg_pos={0 0}, upload=442433]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:40:19.953401942 (504h11m19.824484977s) [upload=442433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:40:19.953408223 (504h11m19.824491258s) [upload=442433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:40:20.252364659 (504h11m20.123447714s) [upload=442433]: Done uploading segments... Dec 26 15:40:20.252393865 (504h11m20.12347691s) [upload=442433]: Upload finished. err: <nil> Duration: 56.39201596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X] duration: 5.001975564s trace: https://jaeger.dev.storj.io/trace/3872e6d9920cd189 FAIL (2025-12-26 15:42:48 UTC) orbiter warning: context deadline exceeded Duration: 5.002007375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.....X.....X.X............XXX....XX...X......X..X..X.X....] duration: 6.139170512s trace: https://jaeger.dev.storj.io/trace/6fca4869d642b9be FAIL (2025-12-26 15:42:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ) context deadline exceeded ======LOGS====== Dec 26 15:38:11.884127603 (504h9m11.755210648s) [upload=442407]: Starting upload Dec 26 15:38:11.884220320 (504h9m11.755303375s) [upload=442407]: Done starting upload Dec 26 15:38:11.884476737 (504h9m11.755559773s) [upload=442407]: Uploading segments... Dec 26 15:38:11.884552192 (504h9m11.755635237s) [upload=442407]: Got next segment. Inline: false Dec 26 15:38:11.884592018 (504h9m11.755675063s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:38:11.884598049 (504h9m11.755681094s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:38:12.134094724 (504h9m12.005177759s) [seg_pos={0 0}, upload=442407]: Begin upload segment... Dec 26 15:38:12.134151552 (504h9m12.005234597s) [seg_pos={0 0}, upload=442407]: Done begin upload segment. Dec 26 15:38:12.134183844 (504h9m12.005266879s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134257975 (504h9m12.00534102s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134939473 (504h9m12.006022518s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135488959 (504h9m12.006572004s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135931852 (504h9m12.007014897s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136399252 (504h9m12.007482298s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136917600 (504h9m12.008000645s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.138384252 (504h9m12.009467297s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139126175 (504h9m12.01020921s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139807833 (504h9m12.010890868s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141360208 (504h9m12.012443254s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141717439 (504h9m12.012800485s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.143312085 (504h9m12.01439512s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.144758760 (504h9m12.015841795s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.146963618 (504h9m12.018046653s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.149176833 (504h9m12.020259878s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:13.860029024 (504h9m13.731112069s) [upload=442407]: Next returned nil segment Dec 26 15:38:13.860051176 (504h9m13.731134221s) [upload=442407]: Waiting for error group managing segments... Dec 26 15:38:14.140188417 (504h9m14.011271452s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.149246274 (504h9m14.020329319s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.150448424 (504h9m14.021531499s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.155601316 (504h9m14.026684361s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.157887398 (504h9m14.028970433s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.171654053 (504h9m14.042737098s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174269793 (504h9m14.045352828s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174722906 (504h9m14.045805941s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.399247390 (504h9m14.270330425s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.418217913 (504h9m14.289300948s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.524304791 (504h9m14.395387836s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.585548651 (504h9m14.456631696s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615716845 (504h9m14.48679988s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615737164 (504h9m14.4868202s) [seg_pos={0 0}, upload=442407]: Segment reached optimal threshold of 13 pieces. Dec 26 15:38:14.615911667 (504h9m14.486994702s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection Dec 26 15:38:14.615917758 (504h9m14.487000793s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616222659 (504h9m14.487305694s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616229632 (504h9m14.487312677s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616336716 (504h9m14.487419751s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616342036 (504h9m14.487425081s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616356093 (504h9m14.487439128s) [seg_pos={0 0}, upload=442407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:38:14.616417860 (504h9m14.487500895s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:38:14.616422590 (504h9m14.487505625s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:38:14.917908311 (504h9m14.788991356s) [upload=442407]: Done uploading segments... Dec 26 15:38:14.917924912 (504h9m14.789007957s) [upload=442407]: Upload finished. err: <nil> Duration: 6.139197422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..X.X...XX.XXXX.X..................X..........X.XX.X....X.] duration: 33.153019135s trace: https://jaeger.dev.storj.io/trace/dfc94eb2ec733dd FAIL (2025-12-26 15:41:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.079395532 (504h11m16.950478577s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.081581986 (504h11m16.952665031s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082159214 (504h11m16.953242259s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082917899 (504h11m16.954000944s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083135794 (504h11m16.954218839s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083289507 (504h11m16.954372542s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083912283 (504h11m16.954995318s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.084313877 (504h11m16.955396922s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085026795 (504h11m16.95610984s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085578926 (504h11m16.956661971s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085601299 (504h11m16.956684344s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086213805 (504h11m16.95729685s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086714428 (504h11m16.957797473s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.087503671 (504h11m16.958586727s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:18.448801149 (504h11m18.319884194s) [upload=442433]: Next returned nil segment Dec 26 15:40:18.448816528 (504h11m18.319899563s) [upload=442433]: Waiting for error group managing segments... Dec 26 15:40:18.795258594 (504h11m18.666341629s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.980610968 (504h11m18.851694013s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.992478436 (504h11m18.863561481s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.168471206 (504h11m19.039554242s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.198368747 (504h11m19.069451792s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.230854365 (504h11m19.1019374s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.291407819 (504h11m19.162490875s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.305940752 (504h11m19.177023787s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.412658281 (504h11m19.283741326s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.416084726 (504h11m19.287167771s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.863940842 (504h11m19.735023887s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.872450075 (504h11m19.74353312s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.918830151 (504h11m19.789913196s) [noise=true , address=145.79.227.34:4924 , piece_id=TTVUB2HO2U4LYY6RCT7FOHX4KQ3BBJRJF6UXXAZIPILINMMGUI3A, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952518840 (504h11m19.823601875s) [noise=true , address=145.79.227.30:4779 , piece_id=HM64K7CTAIV6HYTM64KTSXDFP5GZZVHCN3D6WY5ZH6DQ7XZDHHVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952540902 (504h11m19.823623947s) [seg_pos={0 0}, upload=442433]: Segment reached optimal threshold of 14 pieces. Dec 26 15:40:19.952728510 (504h11m19.823811545s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection Dec 26 15:40:19.952735372 (504h11m19.823818407s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952923771 (504h11m19.824006806s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection Dec 26 15:40:19.952958006 (504h11m19.824041051s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952976081 (504h11m19.824059126s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.952983415 (504h11m19.82406646s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953228752 (504h11m19.824311787s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T4C5mqpUTmi8sZq38KuFakx6Xh6FTVBn5jCFqFHr5GAyBBDaY): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.953235955 (504h11m19.824319s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953257617 (504h11m19.824340662s) [seg_pos={0 0}, upload=442433]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:40:19.953401942 (504h11m19.824484977s) [upload=442433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:40:19.953408223 (504h11m19.824491258s) [upload=442433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:40:20.252364659 (504h11m20.123447714s) [upload=442433]: Done uploading segments... Dec 26 15:40:20.252393865 (504h11m20.12347691s) [upload=442433]: Upload finished. err: <nil> Duration: 33.153045354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...] duration: 5.001095779s trace: https://jaeger.dev.storj.io/trace/513147d04c107b8 FAIL (2025-12-26 15:41:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00111646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X.....X.....X.X............XXX....XX...X......X..X..X.X...] duration: 5.857210782s trace: https://jaeger.dev.storj.io/trace/6fca4869d642b9be FAIL (2025-12-26 15:41:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ) context deadline exceeded ======LOGS====== Dec 26 15:38:11.884127603 (504h9m11.755210648s) [upload=442407]: Starting upload Dec 26 15:38:11.884220320 (504h9m11.755303375s) [upload=442407]: Done starting upload Dec 26 15:38:11.884476737 (504h9m11.755559773s) [upload=442407]: Uploading segments... Dec 26 15:38:11.884552192 (504h9m11.755635237s) [upload=442407]: Got next segment. Inline: false Dec 26 15:38:11.884592018 (504h9m11.755675063s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:38:11.884598049 (504h9m11.755681094s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:38:12.134094724 (504h9m12.005177759s) [seg_pos={0 0}, upload=442407]: Begin upload segment... Dec 26 15:38:12.134151552 (504h9m12.005234597s) [seg_pos={0 0}, upload=442407]: Done begin upload segment. Dec 26 15:38:12.134183844 (504h9m12.005266879s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134257975 (504h9m12.00534102s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134939473 (504h9m12.006022518s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135488959 (504h9m12.006572004s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135931852 (504h9m12.007014897s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136399252 (504h9m12.007482298s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136917600 (504h9m12.008000645s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.138384252 (504h9m12.009467297s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139126175 (504h9m12.01020921s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139807833 (504h9m12.010890868s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141360208 (504h9m12.012443254s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141717439 (504h9m12.012800485s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.143312085 (504h9m12.01439512s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.144758760 (504h9m12.015841795s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.146963618 (504h9m12.018046653s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.149176833 (504h9m12.020259878s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:13.860029024 (504h9m13.731112069s) [upload=442407]: Next returned nil segment Dec 26 15:38:13.860051176 (504h9m13.731134221s) [upload=442407]: Waiting for error group managing segments... Dec 26 15:38:14.140188417 (504h9m14.011271452s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.149246274 (504h9m14.020329319s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.150448424 (504h9m14.021531499s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.155601316 (504h9m14.026684361s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.157887398 (504h9m14.028970433s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.171654053 (504h9m14.042737098s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174269793 (504h9m14.045352828s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174722906 (504h9m14.045805941s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.399247390 (504h9m14.270330425s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.418217913 (504h9m14.289300948s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.524304791 (504h9m14.395387836s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.585548651 (504h9m14.456631696s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615716845 (504h9m14.48679988s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615737164 (504h9m14.4868202s) [seg_pos={0 0}, upload=442407]: Segment reached optimal threshold of 13 pieces. Dec 26 15:38:14.615911667 (504h9m14.486994702s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection Dec 26 15:38:14.615917758 (504h9m14.487000793s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616222659 (504h9m14.487305694s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616229632 (504h9m14.487312677s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616336716 (504h9m14.487419751s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616342036 (504h9m14.487425081s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616356093 (504h9m14.487439128s) [seg_pos={0 0}, upload=442407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:38:14.616417860 (504h9m14.487500895s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:38:14.616422590 (504h9m14.487505625s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:38:14.917908311 (504h9m14.788991356s) [upload=442407]: Done uploading segments... Dec 26 15:38:14.917924912 (504h9m14.789007957s) [upload=442407]: Upload finished. err: <nil> Duration: 5.85724133s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X..X.X...XX.XXXX.X..................X..........X.XX.X....X] duration: 1m0.003144498s trace: https://jaeger.dev.storj.io/trace/dfc94eb2ec733dd FAIL (2025-12-26 15:41:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.079395532 (504h11m16.950478577s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.081581986 (504h11m16.952665031s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082159214 (504h11m16.953242259s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.082917899 (504h11m16.954000944s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083135794 (504h11m16.954218839s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083289507 (504h11m16.954372542s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.083912283 (504h11m16.954995318s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.084313877 (504h11m16.955396922s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085026795 (504h11m16.95610984s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085578926 (504h11m16.956661971s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.085601299 (504h11m16.956684344s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086213805 (504h11m16.95729685s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.086714428 (504h11m16.957797473s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:17.087503671 (504h11m16.958586727s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Uploading piece... Dec 26 15:40:18.448801149 (504h11m18.319884194s) [upload=442433]: Next returned nil segment Dec 26 15:40:18.448816528 (504h11m18.319899563s) [upload=442433]: Waiting for error group managing segments... Dec 26 15:40:18.795258594 (504h11m18.666341629s) [noise=true , address=193.149.164.143:4695 , piece_id=HMX7EDARATQFMB63RPTIGCYX73L47ZHW37NLOEKFWBCYDBGCU5EQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.980610968 (504h11m18.851694013s) [noise=true , address=145.79.227.43:4164 , piece_id=F2QBJKFPK2HRXKIL52ZBTNXTST6VYLUW46TZK2ZTRULQULPS6ROQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:18.992478436 (504h11m18.863561481s) [noise=true , address=145.79.227.50:4387 , piece_id=IVQUGUMXBPO4GLYAFOAT5YO7S56ZICMY426IAS3QQ24Q3YGPP4IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.168471206 (504h11m19.039554242s) [noise=true , address=45.91.116.10:4115 , piece_id=ZZT54H3LTMVRHZ7RGNQLHDTS46DH76A3TG2PPELU6A4OAFWQNKEA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.198368747 (504h11m19.069451792s) [noise=true , address=45.91.116.12:4236 , piece_id=A26CVWM4GLF7W3SZSSE25DDZ2YLGALBRDDF6DBIUOKZCWOE4PTVQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.230854365 (504h11m19.1019374s) [noise=true , address=77.90.31.21:4888 , piece_id=23VMCTYVHFUJX5IDJ4EVZIRTE3VNWGNFKN7YJF2MK6VNG4OQN3IA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.291407819 (504h11m19.162490875s) [noise=true , address=77.90.31.13:4631 , piece_id=PIMETWFX2TSCZC4UCW5VYTUD4RGGJYSHXKMBMKSKWVXFDAAFNDQQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.305940752 (504h11m19.177023787s) [noise=true , address=45.91.116.9:4291 , piece_id=2AEIX5RVYUEJGHVTWISQ5W3BEH6PX7XZ7UNIYOTGINZU2CFXUASQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.412658281 (504h11m19.283741326s) [noise=true , address=77.90.31.16:4761 , piece_id=HX37SWDZ5JU2PZPEWN44LLYLB5DIXIDHTHFAYRRN3M2X4ZCZBOSQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.416084726 (504h11m19.287167771s) [noise=true , address=77.90.31.15:4703 , piece_id=UYKQVLVTFKBPOR4QMAFQ3AFUGZATZE5RFU64MW5USKF5CMV2YHJA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.863940842 (504h11m19.735023887s) [noise=true , address=145.79.227.44:4206 , piece_id=HYFAWB224U3PJDRVRSXXKEVJP2XQBZ3EJRTH6UE5LZBEXFMXBOMA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.872450075 (504h11m19.74353312s) [noise=true , address=77.90.31.18:4815 , piece_id=3NV4Y6I5DIINC2XVUMCC5CZ77IGXOAJOLDKYC7BKQBC4KQ2VNXBA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.918830151 (504h11m19.789913196s) [noise=true , address=145.79.227.34:4924 , piece_id=TTVUB2HO2U4LYY6RCT7FOHX4KQ3BBJRJF6UXXAZIPILINMMGUI3A, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952518840 (504h11m19.823601875s) [noise=true , address=145.79.227.30:4779 , piece_id=HM64K7CTAIV6HYTM64KTSXDFP5GZZVHCN3D6WY5ZH6DQ7XZDHHVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: <nil> Dec 26 15:40:19.952540902 (504h11m19.823623947s) [seg_pos={0 0}, upload=442433]: Segment reached optimal threshold of 14 pieces. Dec 26 15:40:19.952728510 (504h11m19.823811545s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51496->64.49.13.169:4707: use of closed network connection Dec 26 15:40:19.952735372 (504h11m19.823818407s) [noise=true , address=64.49.13.169:4707 , piece_id=A6YCQZZQY5CBLSF4XIBUTZWVVQEC3DAJV5LBZMTFO2GKW26I5STA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952923771 (504h11m19.824006806s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->77.90.31.11:4558: use of closed network connection Dec 26 15:40:19.952958006 (504h11m19.824041051s) [noise=true , address=77.90.31.11:4558 , piece_id=HS7TBUFUCMM7TQZO357PH6QPRQP72C45W72COOC5EEZFX3KECZWA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.952976081 (504h11m19.824059126s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.952983415 (504h11m19.82406646s) [noise=true , address=145.79.227.40:4058 , piece_id=RJMQ5DRHTNSJD4ZJD6GIYNHFYCZJ6QY2AYJOGTKAJJEBGDPLXQVA, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953228752 (504h11m19.824311787s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T4C5mqpUTmi8sZq38KuFakx6Xh6FTVBn5jCFqFHr5GAyBBDaY): protocol: expected piece hash; context canceled; EOF Dec 26 15:40:19.953235955 (504h11m19.824319s) [noise=true , address=145.79.227.24:4587 , piece_id=P6SBNUP2V4JAMA2JCLRX2LWUSOXWQ54C4NL5MO5OHMELNIRWXJCQ, seg_pos={0 0}, upload=442433]: Long tail cancellation due to optimal threshold Dec 26 15:40:19.953257617 (504h11m19.824340662s) [seg_pos={0 0}, upload=442433]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:40:19.953401942 (504h11m19.824484977s) [upload=442433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:40:19.953408223 (504h11m19.824491258s) [upload=442433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:40:20.252364659 (504h11m20.123447714s) [upload=442433]: Done uploading segments... Dec 26 15:40:20.252393865 (504h11m20.12347691s) [upload=442433]: Upload finished. err: <nil> Duration: 1m0.003254468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX..] duration: 5.001463379s trace: https://jaeger.dev.storj.io/trace/513147d04c107b8 FAIL (2025-12-26 15:41:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001481985s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X.....X.....X.X............XXX....XX...X......X..X..X.X..] duration: 5.784851978s trace: https://jaeger.dev.storj.io/trace/6fca4869d642b9be FAIL (2025-12-26 15:40:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ) context deadline exceeded ======LOGS====== Dec 26 15:38:11.884127603 (504h9m11.755210648s) [upload=442407]: Starting upload Dec 26 15:38:11.884220320 (504h9m11.755303375s) [upload=442407]: Done starting upload Dec 26 15:38:11.884476737 (504h9m11.755559773s) [upload=442407]: Uploading segments... Dec 26 15:38:11.884552192 (504h9m11.755635237s) [upload=442407]: Got next segment. Inline: false Dec 26 15:38:11.884592018 (504h9m11.755675063s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:38:11.884598049 (504h9m11.755681094s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:38:12.134094724 (504h9m12.005177759s) [seg_pos={0 0}, upload=442407]: Begin upload segment... Dec 26 15:38:12.134151552 (504h9m12.005234597s) [seg_pos={0 0}, upload=442407]: Done begin upload segment. Dec 26 15:38:12.134183844 (504h9m12.005266879s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134257975 (504h9m12.00534102s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134939473 (504h9m12.006022518s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135488959 (504h9m12.006572004s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135931852 (504h9m12.007014897s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136399252 (504h9m12.007482298s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136917600 (504h9m12.008000645s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.138384252 (504h9m12.009467297s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139126175 (504h9m12.01020921s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139807833 (504h9m12.010890868s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141360208 (504h9m12.012443254s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141717439 (504h9m12.012800485s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.143312085 (504h9m12.01439512s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.144758760 (504h9m12.015841795s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.146963618 (504h9m12.018046653s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.149176833 (504h9m12.020259878s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:13.860029024 (504h9m13.731112069s) [upload=442407]: Next returned nil segment Dec 26 15:38:13.860051176 (504h9m13.731134221s) [upload=442407]: Waiting for error group managing segments... Dec 26 15:38:14.140188417 (504h9m14.011271452s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.149246274 (504h9m14.020329319s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.150448424 (504h9m14.021531499s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.155601316 (504h9m14.026684361s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.157887398 (504h9m14.028970433s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.171654053 (504h9m14.042737098s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174269793 (504h9m14.045352828s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174722906 (504h9m14.045805941s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.399247390 (504h9m14.270330425s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.418217913 (504h9m14.289300948s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.524304791 (504h9m14.395387836s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.585548651 (504h9m14.456631696s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615716845 (504h9m14.48679988s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615737164 (504h9m14.4868202s) [seg_pos={0 0}, upload=442407]: Segment reached optimal threshold of 13 pieces. Dec 26 15:38:14.615911667 (504h9m14.486994702s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection Dec 26 15:38:14.615917758 (504h9m14.487000793s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616222659 (504h9m14.487305694s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616229632 (504h9m14.487312677s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616336716 (504h9m14.487419751s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616342036 (504h9m14.487425081s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616356093 (504h9m14.487439128s) [seg_pos={0 0}, upload=442407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:38:14.616417860 (504h9m14.487500895s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:38:14.616422590 (504h9m14.487505625s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:38:14.917908311 (504h9m14.788991356s) [upload=442407]: Done uploading segments... Dec 26 15:38:14.917924912 (504h9m14.789007957s) [upload=442407]: Upload finished. err: <nil> Duration: 5.784880832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X..X.X...XX.XXXX.X..................X..........X.XX.X....] duration: 31.092074625s trace: https://jaeger.dev.storj.io/trace/18fbdbd65f37dab6 FAIL (2025-12-26 15:40:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ) context deadline exceeded; piecestore: (Node ID: 12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, Piece ID: IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: 5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA) context deadline exceeded ======LOGS====== 22141s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119456000 (504h6m34.990539035s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119767683 (504h6m34.990850728s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120122229 (504h6m34.991205264s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120490921 (504h6m34.991573956s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120909187 (504h6m34.991992222s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121373372 (504h6m34.992456407s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121393289 (504h6m34.992476334s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121900044 (504h6m34.992983089s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122331596 (504h6m34.993414631s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122778266 (504h6m34.993861311s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123296363 (504h6m34.994379408s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123820291 (504h6m34.994903336s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124272331 (504h6m34.995355376s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124440031 (504h6m34.995523066s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:36.442690439 (504h6m36.313773484s) [upload=442372]: Next returned nil segment Dec 26 15:35:36.442714626 (504h6m36.313797661s) [upload=442372]: Waiting for error group managing segments... Dec 26 15:35:36.799517290 (504h6m36.670600325s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.164491694 (504h6m37.035574739s) [noise=true , address=77.90.31.12:4616 , piece_id=VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.169500051 (504h6m37.040583096s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.222379363 (504h6m37.093462398s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.230139129 (504h6m37.101222174s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.335458906 (504h6m37.206541951s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.368208557 (504h6m37.239291592s) [noise=true , address=77.90.31.16:4757 , piece_id=KTP5KVZPWXG4PSX6DBXGZHWLAKCJYSXBF5OXOQC37DWISNIYGATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.428078270 (504h6m37.299161305s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.513975203 (504h6m37.385058238s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.835023545 (504h6m37.70610659s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.908251443 (504h6m37.779334489s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.944654971 (504h6m37.815738016s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.996043885 (504h6m37.867126931s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217717732 (504h6m38.088800777s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217742618 (504h6m38.088825653s) [seg_pos={0 0}, upload=442372]: Segment reached optimal threshold of 14 pieces. Dec 26 15:35:38.218057368 (504h6m38.089140413s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218068630 (504h6m38.089151675s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218083247 (504h6m38.089166282s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection Dec 26 15:35:38.218099709 (504h6m38.089182744s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218246418 (504h6m38.089329453s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218256748 (504h6m38.089339783s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218418897 (504h6m38.089501932s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection Dec 26 15:35:38.218427152 (504h6m38.089510187s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218594601 (504h6m38.089677636s) [seg_pos={0 0}, upload=442372]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:35:38.218708177 (504h6m38.089791212s) [upload=442372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:35:38.218718356 (504h6m38.089801391s) [upload=442372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:35:38.513584834 (504h6m38.38466787s) [upload=442372]: Done uploading segments... Dec 26 15:35:38.513609711 (504h6m38.384692756s) [upload=442372]: Upload finished. err: <nil> Duration: 31.092099605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX.] duration: 3.952665825s trace: https://jaeger.dev.storj.io/trace/513147d04c107b8 FAIL (2025-12-26 15:39:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.95269442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..X..X.X...XX.XXXX.X..................X..........X.XX.X...] duration: 50.971664335s trace: https://jaeger.dev.storj.io/trace/18fbdbd65f37dab6 FAIL (2025-12-26 15:39:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ) context deadline exceeded; piecestore: (Node ID: 12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, Piece ID: IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: 5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA) context deadline exceeded ======LOGS====== 22141s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119456000 (504h6m34.990539035s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119767683 (504h6m34.990850728s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120122229 (504h6m34.991205264s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120490921 (504h6m34.991573956s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120909187 (504h6m34.991992222s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121373372 (504h6m34.992456407s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121393289 (504h6m34.992476334s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121900044 (504h6m34.992983089s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122331596 (504h6m34.993414631s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122778266 (504h6m34.993861311s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123296363 (504h6m34.994379408s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123820291 (504h6m34.994903336s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124272331 (504h6m34.995355376s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124440031 (504h6m34.995523066s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:36.442690439 (504h6m36.313773484s) [upload=442372]: Next returned nil segment Dec 26 15:35:36.442714626 (504h6m36.313797661s) [upload=442372]: Waiting for error group managing segments... Dec 26 15:35:36.799517290 (504h6m36.670600325s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.164491694 (504h6m37.035574739s) [noise=true , address=77.90.31.12:4616 , piece_id=VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.169500051 (504h6m37.040583096s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.222379363 (504h6m37.093462398s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.230139129 (504h6m37.101222174s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.335458906 (504h6m37.206541951s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.368208557 (504h6m37.239291592s) [noise=true , address=77.90.31.16:4757 , piece_id=KTP5KVZPWXG4PSX6DBXGZHWLAKCJYSXBF5OXOQC37DWISNIYGATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.428078270 (504h6m37.299161305s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.513975203 (504h6m37.385058238s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.835023545 (504h6m37.70610659s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.908251443 (504h6m37.779334489s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.944654971 (504h6m37.815738016s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.996043885 (504h6m37.867126931s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217717732 (504h6m38.088800777s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217742618 (504h6m38.088825653s) [seg_pos={0 0}, upload=442372]: Segment reached optimal threshold of 14 pieces. Dec 26 15:35:38.218057368 (504h6m38.089140413s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218068630 (504h6m38.089151675s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218083247 (504h6m38.089166282s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection Dec 26 15:35:38.218099709 (504h6m38.089182744s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218246418 (504h6m38.089329453s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218256748 (504h6m38.089339783s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218418897 (504h6m38.089501932s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection Dec 26 15:35:38.218427152 (504h6m38.089510187s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218594601 (504h6m38.089677636s) [seg_pos={0 0}, upload=442372]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:35:38.218708177 (504h6m38.089791212s) [upload=442372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:35:38.218718356 (504h6m38.089801391s) [upload=442372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:35:38.513584834 (504h6m38.38466787s) [upload=442372]: Done uploading segments... Dec 26 15:35:38.513609711 (504h6m38.384692756s) [upload=442372]: Upload finished. err: <nil> Duration: 50.971678511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.X.....X.....X.X............XXX....XX...X......X..X..X.X.] duration: 5.833869025s trace: https://jaeger.dev.storj.io/trace/6fca4869d642b9be FAIL (2025-12-26 15:39:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ) context deadline exceeded ======LOGS====== Dec 26 15:38:11.884127603 (504h9m11.755210648s) [upload=442407]: Starting upload Dec 26 15:38:11.884220320 (504h9m11.755303375s) [upload=442407]: Done starting upload Dec 26 15:38:11.884476737 (504h9m11.755559773s) [upload=442407]: Uploading segments... Dec 26 15:38:11.884552192 (504h9m11.755635237s) [upload=442407]: Got next segment. Inline: false Dec 26 15:38:11.884592018 (504h9m11.755675063s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:38:11.884598049 (504h9m11.755681094s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:38:12.134094724 (504h9m12.005177759s) [seg_pos={0 0}, upload=442407]: Begin upload segment... Dec 26 15:38:12.134151552 (504h9m12.005234597s) [seg_pos={0 0}, upload=442407]: Done begin upload segment. Dec 26 15:38:12.134183844 (504h9m12.005266879s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134257975 (504h9m12.00534102s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134939473 (504h9m12.006022518s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135488959 (504h9m12.006572004s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135931852 (504h9m12.007014897s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136399252 (504h9m12.007482298s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136917600 (504h9m12.008000645s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.138384252 (504h9m12.009467297s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139126175 (504h9m12.01020921s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139807833 (504h9m12.010890868s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141360208 (504h9m12.012443254s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141717439 (504h9m12.012800485s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.143312085 (504h9m12.01439512s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.144758760 (504h9m12.015841795s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.146963618 (504h9m12.018046653s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.149176833 (504h9m12.020259878s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:13.860029024 (504h9m13.731112069s) [upload=442407]: Next returned nil segment Dec 26 15:38:13.860051176 (504h9m13.731134221s) [upload=442407]: Waiting for error group managing segments... Dec 26 15:38:14.140188417 (504h9m14.011271452s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.149246274 (504h9m14.020329319s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.150448424 (504h9m14.021531499s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.155601316 (504h9m14.026684361s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.157887398 (504h9m14.028970433s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.171654053 (504h9m14.042737098s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174269793 (504h9m14.045352828s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174722906 (504h9m14.045805941s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.399247390 (504h9m14.270330425s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.418217913 (504h9m14.289300948s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.524304791 (504h9m14.395387836s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.585548651 (504h9m14.456631696s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615716845 (504h9m14.48679988s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615737164 (504h9m14.4868202s) [seg_pos={0 0}, upload=442407]: Segment reached optimal threshold of 13 pieces. Dec 26 15:38:14.615911667 (504h9m14.486994702s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection Dec 26 15:38:14.615917758 (504h9m14.487000793s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616222659 (504h9m14.487305694s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616229632 (504h9m14.487312677s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616336716 (504h9m14.487419751s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616342036 (504h9m14.487425081s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616356093 (504h9m14.487439128s) [seg_pos={0 0}, upload=442407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:38:14.616417860 (504h9m14.487500895s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:38:14.616422590 (504h9m14.487505625s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:38:14.917908311 (504h9m14.788991356s) [upload=442407]: Done uploading segments... Dec 26 15:38:14.917924912 (504h9m14.789007957s) [upload=442407]: Upload finished. err: <nil> Duration: 5.833897389s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX] duration: 5.001461014s trace: https://jaeger.dev.storj.io/trace/513147d04c107b8 FAIL (2025-12-26 15:39:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001487035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X.X.....X.....X.X............XXX....XX...X......X..X..X.X] duration: 30.003107593s trace: https://jaeger.dev.storj.io/trace/6fca4869d642b9be FAIL (2025-12-26 15:38:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ) context deadline exceeded ======LOGS====== Dec 26 15:38:11.884127603 (504h9m11.755210648s) [upload=442407]: Starting upload Dec 26 15:38:11.884220320 (504h9m11.755303375s) [upload=442407]: Done starting upload Dec 26 15:38:11.884476737 (504h9m11.755559773s) [upload=442407]: Uploading segments... Dec 26 15:38:11.884552192 (504h9m11.755635237s) [upload=442407]: Got next segment. Inline: false Dec 26 15:38:11.884592018 (504h9m11.755675063s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:38:11.884598049 (504h9m11.755681094s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:38:12.134094724 (504h9m12.005177759s) [seg_pos={0 0}, upload=442407]: Begin upload segment... Dec 26 15:38:12.134151552 (504h9m12.005234597s) [seg_pos={0 0}, upload=442407]: Done begin upload segment. Dec 26 15:38:12.134183844 (504h9m12.005266879s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134257975 (504h9m12.00534102s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.134939473 (504h9m12.006022518s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135488959 (504h9m12.006572004s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.135931852 (504h9m12.007014897s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136399252 (504h9m12.007482298s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.136917600 (504h9m12.008000645s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.138384252 (504h9m12.009467297s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139126175 (504h9m12.01020921s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.139807833 (504h9m12.010890868s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141360208 (504h9m12.012443254s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.141717439 (504h9m12.012800485s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.143312085 (504h9m12.01439512s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.144758760 (504h9m12.015841795s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.146963618 (504h9m12.018046653s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:12.149176833 (504h9m12.020259878s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Uploading piece... Dec 26 15:38:13.860029024 (504h9m13.731112069s) [upload=442407]: Next returned nil segment Dec 26 15:38:13.860051176 (504h9m13.731134221s) [upload=442407]: Waiting for error group managing segments... Dec 26 15:38:14.140188417 (504h9m14.011271452s) [noise=true , address=145.79.227.11:4730 , piece_id=2VVNJF6D7FE44BLAYR2LFXVRE5R4SQCI6KEVCBBK7L7UXK4OZFKA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.149246274 (504h9m14.020329319s) [noise=true , address=145.79.227.35:4482 , piece_id=FHOKFGRC7LNJEGZ3J6B5HVDQFRW6DJS5FTNVDZCR5AXTL5MJS2WQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.150448424 (504h9m14.021531499s) [noise=true , address=145.79.227.45:4229 , piece_id=MWDECMFKJRAXWKRUJ737YMQVK4JRWI4UJ4YJ4IQLY6CTN3KZ7A4Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.155601316 (504h9m14.026684361s) [noise=true , address=145.79.227.31:4799 , piece_id=RKERYPMCPYU63LJH3NRTSDC2NR53DJAF6YMIJKT4NCBJ7WLS6ZLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.157887398 (504h9m14.028970433s) [noise=true , address=145.79.227.50:4375 , piece_id=Z666UAM3PR7YUJBUDO63E3SSEZXD6ZGHIS6QBRPN3OWPIDDMOYVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.171654053 (504h9m14.042737098s) [noise=true , address=145.79.227.12:4767 , piece_id=IZZ6U66W62I6TOR2WFGZWXV6WJJDNSSJX5ZJH43H47WO4WSQ73WA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174269793 (504h9m14.045352828s) [noise=true , address=145.79.227.36:4518 , piece_id=XCZF6DUAXRX7V3NTM34OWWCEAKMBGSSTEDKS2GS6ZQO3HTJFBXNQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.174722906 (504h9m14.045805941s) [noise=true , address=145.79.227.46:4601 , piece_id=TPW2TC7KZGOPMS2KSXJ2YGC5HK5ER5M7BAVXP2OPWR6RDCNKKSJA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.399247390 (504h9m14.270330425s) [noise=true , address=45.91.116.11:4119 , piece_id=O6QT6YXYK7HMDEWKWZ4H2KG3BRKTFJKK4DGN7SK4Y57TPG2CD33A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.418217913 (504h9m14.289300948s) [noise=true , address=45.91.116.13:4156 , piece_id=FWQRGZCEWUHOV6XB6EZNYVIDEP4JS6PBUWBA34QD23LWQDNMUX6A, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.524304791 (504h9m14.395387836s) [noise=true , address=45.91.116.9:4264 , piece_id=VMSSGHPFNYB5ZELJDDQNHK5NBH22T2Q3AILHUPSKA2BUPA5UW73Q, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.585548651 (504h9m14.456631696s) [noise=true , address=45.91.116.14:4189 , piece_id=OK7T4633G7YD3TJVNE6STCRWSQXDY4ULTBMZERP24LMMTW67BXIA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615716845 (504h9m14.48679988s) [noise=true , address=45.91.116.7:4011 , piece_id=6VUHPEC6L46YMWM6TVXBBAWHGAFFEQZEV2A5MVP24FW3BUTHJAVQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: <nil> Dec 26 15:38:14.615737164 (504h9m14.4868202s) [seg_pos={0 0}, upload=442407]: Segment reached optimal threshold of 13 pieces. Dec 26 15:38:14.615911667 (504h9m14.486994702s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->45.91.116.6:4976: use of closed network connection Dec 26 15:38:14.615917758 (504h9m14.487000793s) [noise=true , address=45.91.116.6:4976 , piece_id=PIKNCNZCXF2IE7PTYBFVQ23WOVNBVND7TTT33S5WCQPMSWQI7IVA, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616222659 (504h9m14.487305694s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616229632 (504h9m14.487312677s) [noise=true , address=77.90.31.17:4773 , piece_id=MTPGYMCXJNUVARCB2JOUZTV4S72AGEUVZRFOVUPXRCAL3BI7SGLQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616336716 (504h9m14.487419751s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 15:38:14.616342036 (504h9m14.487425081s) [noise=true , address=45.91.116.5:4298 , piece_id=OWF2XZJBTTRDJSPKF73QDUS2DKWKK2LX4KW5RHXJYEJUOA3FANGQ, seg_pos={0 0}, upload=442407]: Long tail cancellation due to optimal threshold Dec 26 15:38:14.616356093 (504h9m14.487439128s) [seg_pos={0 0}, upload=442407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:38:14.616417860 (504h9m14.487500895s) [upload=442407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:38:14.616422590 (504h9m14.487505625s) [upload=442407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:38:14.917908311 (504h9m14.788991356s) [upload=442407]: Done uploading segments... Dec 26 15:38:14.917924912 (504h9m14.789007957s) [upload=442407]: Upload finished. err: <nil> Duration: 30.003119575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X..X..X.X...XX.XXXX.X..................X..........X.XX.X..] duration: 51.086041087s trace: https://jaeger.dev.storj.io/trace/18fbdbd65f37dab6 FAIL (2025-12-26 15:38:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ) context deadline exceeded; piecestore: (Node ID: 12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, Piece ID: IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: 5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA) context deadline exceeded ======LOGS====== 22141s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119456000 (504h6m34.990539035s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119767683 (504h6m34.990850728s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120122229 (504h6m34.991205264s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120490921 (504h6m34.991573956s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120909187 (504h6m34.991992222s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121373372 (504h6m34.992456407s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121393289 (504h6m34.992476334s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121900044 (504h6m34.992983089s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122331596 (504h6m34.993414631s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122778266 (504h6m34.993861311s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123296363 (504h6m34.994379408s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123820291 (504h6m34.994903336s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124272331 (504h6m34.995355376s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124440031 (504h6m34.995523066s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:36.442690439 (504h6m36.313773484s) [upload=442372]: Next returned nil segment Dec 26 15:35:36.442714626 (504h6m36.313797661s) [upload=442372]: Waiting for error group managing segments... Dec 26 15:35:36.799517290 (504h6m36.670600325s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.164491694 (504h6m37.035574739s) [noise=true , address=77.90.31.12:4616 , piece_id=VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.169500051 (504h6m37.040583096s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.222379363 (504h6m37.093462398s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.230139129 (504h6m37.101222174s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.335458906 (504h6m37.206541951s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.368208557 (504h6m37.239291592s) [noise=true , address=77.90.31.16:4757 , piece_id=KTP5KVZPWXG4PSX6DBXGZHWLAKCJYSXBF5OXOQC37DWISNIYGATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.428078270 (504h6m37.299161305s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.513975203 (504h6m37.385058238s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.835023545 (504h6m37.70610659s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.908251443 (504h6m37.779334489s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.944654971 (504h6m37.815738016s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.996043885 (504h6m37.867126931s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217717732 (504h6m38.088800777s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217742618 (504h6m38.088825653s) [seg_pos={0 0}, upload=442372]: Segment reached optimal threshold of 14 pieces. Dec 26 15:35:38.218057368 (504h6m38.089140413s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218068630 (504h6m38.089151675s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218083247 (504h6m38.089166282s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection Dec 26 15:35:38.218099709 (504h6m38.089182744s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218246418 (504h6m38.089329453s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218256748 (504h6m38.089339783s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218418897 (504h6m38.089501932s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection Dec 26 15:35:38.218427152 (504h6m38.089510187s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218594601 (504h6m38.089677636s) [seg_pos={0 0}, upload=442372]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:35:38.218708177 (504h6m38.089791212s) [upload=442372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:35:38.218718356 (504h6m38.089801391s) [upload=442372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:35:38.513584834 (504h6m38.38466787s) [upload=442372]: Done uploading segments... Dec 26 15:35:38.513609711 (504h6m38.384692756s) [upload=442372]: Upload finished. err: <nil> Duration: 51.086055866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.X] duration: 5.002071277s trace: https://jaeger.dev.storj.io/trace/2ad8067754e22748 FAIL (2025-12-26 15:38:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002100111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X..X..X.X...XX.XXXX.X..................X..........X.XX.X.] duration: 59.672665018s trace: https://jaeger.dev.storj.io/trace/18fbdbd65f37dab6 FAIL (2025-12-26 15:37:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ) context deadline exceeded; piecestore: (Node ID: 12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, Piece ID: IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: 5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA) context deadline exceeded ======LOGS====== 22141s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119456000 (504h6m34.990539035s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119767683 (504h6m34.990850728s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120122229 (504h6m34.991205264s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120490921 (504h6m34.991573956s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120909187 (504h6m34.991992222s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121373372 (504h6m34.992456407s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121393289 (504h6m34.992476334s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121900044 (504h6m34.992983089s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122331596 (504h6m34.993414631s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122778266 (504h6m34.993861311s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123296363 (504h6m34.994379408s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123820291 (504h6m34.994903336s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124272331 (504h6m34.995355376s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124440031 (504h6m34.995523066s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:36.442690439 (504h6m36.313773484s) [upload=442372]: Next returned nil segment Dec 26 15:35:36.442714626 (504h6m36.313797661s) [upload=442372]: Waiting for error group managing segments... Dec 26 15:35:36.799517290 (504h6m36.670600325s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.164491694 (504h6m37.035574739s) [noise=true , address=77.90.31.12:4616 , piece_id=VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.169500051 (504h6m37.040583096s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.222379363 (504h6m37.093462398s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.230139129 (504h6m37.101222174s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.335458906 (504h6m37.206541951s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.368208557 (504h6m37.239291592s) [noise=true , address=77.90.31.16:4757 , piece_id=KTP5KVZPWXG4PSX6DBXGZHWLAKCJYSXBF5OXOQC37DWISNIYGATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.428078270 (504h6m37.299161305s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.513975203 (504h6m37.385058238s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.835023545 (504h6m37.70610659s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.908251443 (504h6m37.779334489s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.944654971 (504h6m37.815738016s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.996043885 (504h6m37.867126931s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217717732 (504h6m38.088800777s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217742618 (504h6m38.088825653s) [seg_pos={0 0}, upload=442372]: Segment reached optimal threshold of 14 pieces. Dec 26 15:35:38.218057368 (504h6m38.089140413s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218068630 (504h6m38.089151675s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218083247 (504h6m38.089166282s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection Dec 26 15:35:38.218099709 (504h6m38.089182744s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218246418 (504h6m38.089329453s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218256748 (504h6m38.089339783s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218418897 (504h6m38.089501932s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection Dec 26 15:35:38.218427152 (504h6m38.089510187s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218594601 (504h6m38.089677636s) [seg_pos={0 0}, upload=442372]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:35:38.218708177 (504h6m38.089791212s) [upload=442372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:35:38.218718356 (504h6m38.089801391s) [upload=442372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:35:38.513584834 (504h6m38.38466787s) [upload=442372]: Done uploading segments... Dec 26 15:35:38.513609711 (504h6m38.384692756s) [upload=442372]: Upload finished. err: <nil> Duration: 59.672689906s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.] duration: 5.000505796s trace: https://jaeger.dev.storj.io/trace/6455ce761026ebfd FAIL (2025-12-26 15:37:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000546935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX..X.X.....X.....X.X............XXX....XX...X......X..X..X.] duration: 6.088628943s trace: https://jaeger.dev.storj.io/trace/7bdd148a321f2ef6 FAIL (2025-12-26 15:37:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EC7SD4RCOCTALJFWXCS7IJJGK72SLOVWEHDGX5QH4KVSSTR4WNLQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: Z6GQDPLUM5D3TBFUNES2E5SVGD5LWCWRFB76BPULF2INUQ4DAMFA) context deadline exceeded ======LOGS====== Dec 26 15:36:59.388555843 (504h7m59.259638878s) [upload=442389]: Starting upload Dec 26 15:36:59.388637278 (504h7m59.259720313s) [upload=442389]: Done starting upload Dec 26 15:36:59.388816119 (504h7m59.259899164s) [upload=442389]: Uploading segments... Dec 26 15:36:59.389092345 (504h7m59.26017539s) [upload=442389]: Got next segment. Inline: false Dec 26 15:36:59.389140496 (504h7m59.260223541s) [upload=442389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:36:59.389147500 (504h7m59.260230545s) [upload=442389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:36:59.638379040 (504h7m59.509462085s) [seg_pos={0 0}, upload=442389]: Begin upload segment... Dec 26 15:36:59.638433143 (504h7m59.509516178s) [seg_pos={0 0}, upload=442389]: Done begin upload segment. Dec 26 15:36:59.638494149 (504h7m59.509577184s) [noise=true , address=145.79.227.51:4410 , piece_id=EWWD6SPZ27UWLXYG5XJNM5YGBRJJI3ULDPLGO4GXO64GJUAIQO7A, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.638505330 (504h7m59.509588365s) [noise=true , address=77.90.31.12:4590 , piece_id=I4YVEPX2FYY5X2XYN2JL5B56XQBD6RPY445OLBS376YOQIXCFQWQ, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.639158975 (504h7m59.51024201s) [noise=true , address=45.91.116.11:4120 , piece_id=Z6GQDPLUM5D3TBFUNES2E5SVGD5LWCWRFB76BPULF2INUQ4DAMFA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.639719031 (504h7m59.510802076s) [noise=true , address=145.79.227.46:4601 , piece_id=GIT43LEIHZQCHTJD6TEUQG5MORR3ACMNVM6DO6KL5MWEL4VAZL4A, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.639771722 (504h7m59.510854757s) [noise=true , address=145.79.227.45:4229 , piece_id=UQRKI64XJWWMSGMF47KT24HQ6H36F3FCW3FNHS2FGRUTUCS7UJTA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640155994 (504h7m59.511239029s) [noise=true , address=145.79.227.16:4909 , piece_id=AXUB5V6YFIPTQSCXABQYGPZCZH3FP635KZX2LIQUKLTA3GV2E4JA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640175110 (504h7m59.511258145s) [noise=true , address=145.79.227.41:4085 , piece_id=SKRC7554EXFUJFEM67JZEGYZNVBCSDNQTEH73NJP3NHGEZBNXCTA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640529695 (504h7m59.51161274s) [noise=true , address=145.79.227.24:4581 , piece_id=GTUIO6HDVB4QB7OCQBV52THCXVLD3XNW265RSECIBFWKCO4FVSRA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640633152 (504h7m59.511716187s) [noise=true , address=45.91.116.12:4225 , piece_id=7C4K6563Y6PEBHIQUR3EG62OERSEMBXCKIKV4GOTXX2JWMIBGVXQ, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640929887 (504h7m59.512012922s) [noise=true , address=77.90.31.19:4842 , piece_id=25AH5JQSXIJWC2TWRYHUSLFROHXLDVZDIU42EPUIW7IXCRDNMRJA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.641465527 (504h7m59.512548562s) [noise=true , address=77.90.31.21:4878 , piece_id=EC7SD4RCOCTALJFWXCS7IJJGK72SLOVWEHDGX5QH4KVSSTR4WNLQ, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.641564465 (504h7m59.5126475s) [noise=true , address=145.79.227.8:4330 , piece_id=IZJAKOAHRK4PG3DVUVKV5PXUFM4SMLAGGL2K6SZJ3IZ5FV7HBJTA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.641924482 (504h7m59.513007517s) [noise=true , address=45.91.116.13:4154 , piece_id=WL4RPQ4GKZWCUHFMJC3XQODIJ3QS3FKBB5GOV4OKCCCYRQ2LQAGA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.642115385 (504h7m59.51319843s) [noise=true , address=145.79.227.11:4731 , piece_id=ML5US3WRCDFA2NV2VU3BLISZFOZ3V2HCK3KJBGUQNAD5S32627TQ, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.642397251 (504h7m59.513480286s) [noise=true , address=145.79.227.39:4013 , piece_id=DBKSPPPEJAZ2FAESVEZ2MNILJ4XQRG62D3RZIPT7ILFIZ7FDCPIA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.642883207 (504h7m59.513966242s) [noise=true , address=145.79.227.52:4446 , piece_id=FFPXLJZ4WSIPSTAYCBSIEJ6FQMHATLYHFQTQCJZUA6F7HJICDCLA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:37:01.354857134 (504h8m1.225940179s) [upload=442389]: Next returned nil segment Dec 26 15:37:01.354892431 (504h8m1.225975496s) [upload=442389]: Waiting for error group managing segments... Dec 26 15:37:01.649219571 (504h8m1.520302616s) [noise=true , address=145.79.227.52:4446 , piece_id=FFPXLJZ4WSIPSTAYCBSIEJ6FQMHATLYHFQTQCJZUA6F7HJICDCLA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.650584511 (504h8m1.521667566s) [noise=true , address=145.79.227.46:4601 , piece_id=GIT43LEIHZQCHTJD6TEUQG5MORR3ACMNVM6DO6KL5MWEL4VAZL4A, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.654761024 (504h8m1.525844069s) [noise=true , address=145.79.227.45:4229 , piece_id=UQRKI64XJWWMSGMF47KT24HQ6H36F3FCW3FNHS2FGRUTUCS7UJTA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.660002704 (504h8m1.531085739s) [noise=true , address=145.79.227.8:4330 , piece_id=IZJAKOAHRK4PG3DVUVKV5PXUFM4SMLAGGL2K6SZJ3IZ5FV7HBJTA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.666938421 (504h8m1.538021466s) [noise=true , address=145.79.227.16:4909 , piece_id=AXUB5V6YFIPTQSCXABQYGPZCZH3FP635KZX2LIQUKLTA3GV2E4JA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.667973933 (504h8m1.539056978s) [noise=true , address=145.79.227.11:4731 , piece_id=ML5US3WRCDFA2NV2VU3BLISZFOZ3V2HCK3KJBGUQNAD5S32627TQ, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.671117097 (504h8m1.542200173s) [noise=true , address=145.79.227.51:4410 , piece_id=EWWD6SPZ27UWLXYG5XJNM5YGBRJJI3ULDPLGO4GXO64GJUAIQO7A, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.676015876 (504h8m1.547098921s) [noise=true , address=145.79.227.39:4013 , piece_id=DBKSPPPEJAZ2FAESVEZ2MNILJ4XQRG62D3RZIPT7ILFIZ7FDCPIA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.678757887 (504h8m1.549840932s) [noise=true , address=145.79.227.41:4085 , piece_id=SKRC7554EXFUJFEM67JZEGYZNVBCSDNQTEH73NJP3NHGEZBNXCTA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.679445677 (504h8m1.550528732s) [noise=true , address=145.79.227.24:4581 , piece_id=GTUIO6HDVB4QB7OCQBV52THCXVLD3XNW265RSECIBFWKCO4FVSRA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.933941902 (504h8m1.805024947s) [noise=true , address=45.91.116.11:4120 , piece_id=Z6GQDPLUM5D3TBFUNES2E5SVGD5LWCWRFB76BPULF2INUQ4DAMFA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.999451834 (504h8m1.87053487s) [noise=true , address=77.90.31.21:4878 , piece_id=EC7SD4RCOCTALJFWXCS7IJJGK72SLOVWEHDGX5QH4KVSSTR4WNLQ, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:02.786685434 (504h8m2.65776848s) [noise=true , address=77.90.31.19:4842 , piece_id=25AH5JQSXIJWC2TWRYHUSLFROHXLDVZDIU42EPUIW7IXCRDNMRJA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:02.786708148 (504h8m2.657791183s) [seg_pos={0 0}, upload=442389]: Segment reached optimal threshold of 13 pieces. Dec 26 15:37:02.787036433 (504h8m2.658119478s) [noise=true , address=45.91.116.12:4225 , piece_id=7C4K6563Y6PEBHIQUR3EG62OERSEMBXCKIKV4GOTXX2JWMIBGVXQ, seg_pos={0 0}, upload=442389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 15:37:02.787044568 (504h8m2.658127613s) [noise=true , address=45.91.116.12:4225 , piece_id=7C4K6563Y6PEBHIQUR3EG62OERSEMBXCKIKV4GOTXX2JWMIBGVXQ, seg_pos={0 0}, upload=442389]: Long tail cancellation due to optimal threshold Dec 26 15:37:02.787282050 (504h8m2.658365095s) [noise=true , address=45.91.116.13:4154 , piece_id=WL4RPQ4GKZWCUHFMJC3XQODIJ3QS3FKBB5GOV4OKCCCYRQ2LQAGA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 15:37:02.787292561 (504h8m2.658375596s) [noise=true , address=45.91.116.13:4154 , piece_id=WL4RPQ4GKZWCUHFMJC3XQODIJ3QS3FKBB5GOV4OKCCCYRQ2LQAGA, seg_pos={0 0}, upload=442389]: Long tail cancellation due to optimal threshold Dec 26 15:37:02.787347155 (504h8m2.6584302s) [noise=true , address=77.90.31.12:4590 , piece_id=I4YVEPX2FYY5X2XYN2JL5B56XQBD6RPY445OLBS376YOQIXCFQWQ, seg_pos={0 0}, upload=442389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: read tcp 188.166.225.170:38990->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:37:02.787363666 (504h8m2.658446712s) [noise=true , address=77.90.31.12:4590 , piece_id=I4YVEPX2FYY5X2XYN2JL5B56XQBD6RPY445OLBS376YOQIXCFQWQ, seg_pos={0 0}, upload=442389]: Long tail cancellation due to optimal threshold Dec 26 15:37:02.787399845 (504h8m2.65848289s) [seg_pos={0 0}, upload=442389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:37:02.787495687 (504h8m2.658578722s) [upload=442389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:37:02.787500858 (504h8m2.658583903s) [upload=442389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:37:03.086377230 (504h8m2.957460295s) [upload=442389]: Done uploading segments... Dec 26 15:37:03.086403470 (504h8m2.957486516s) [upload=442389]: Upload finished. err: <nil> Duration: 6.088646516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX..X.X.....X.....X.X............XXX....XX...X......X..X..X] duration: 30.003570494s trace: https://jaeger.dev.storj.io/trace/7bdd148a321f2ef6 FAIL (2025-12-26 15:37:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EC7SD4RCOCTALJFWXCS7IJJGK72SLOVWEHDGX5QH4KVSSTR4WNLQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: Z6GQDPLUM5D3TBFUNES2E5SVGD5LWCWRFB76BPULF2INUQ4DAMFA) context deadline exceeded ======LOGS====== Dec 26 15:36:59.388555843 (504h7m59.259638878s) [upload=442389]: Starting upload Dec 26 15:36:59.388637278 (504h7m59.259720313s) [upload=442389]: Done starting upload Dec 26 15:36:59.388816119 (504h7m59.259899164s) [upload=442389]: Uploading segments... Dec 26 15:36:59.389092345 (504h7m59.26017539s) [upload=442389]: Got next segment. Inline: false Dec 26 15:36:59.389140496 (504h7m59.260223541s) [upload=442389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:36:59.389147500 (504h7m59.260230545s) [upload=442389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:36:59.638379040 (504h7m59.509462085s) [seg_pos={0 0}, upload=442389]: Begin upload segment... Dec 26 15:36:59.638433143 (504h7m59.509516178s) [seg_pos={0 0}, upload=442389]: Done begin upload segment. Dec 26 15:36:59.638494149 (504h7m59.509577184s) [noise=true , address=145.79.227.51:4410 , piece_id=EWWD6SPZ27UWLXYG5XJNM5YGBRJJI3ULDPLGO4GXO64GJUAIQO7A, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.638505330 (504h7m59.509588365s) [noise=true , address=77.90.31.12:4590 , piece_id=I4YVEPX2FYY5X2XYN2JL5B56XQBD6RPY445OLBS376YOQIXCFQWQ, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.639158975 (504h7m59.51024201s) [noise=true , address=45.91.116.11:4120 , piece_id=Z6GQDPLUM5D3TBFUNES2E5SVGD5LWCWRFB76BPULF2INUQ4DAMFA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.639719031 (504h7m59.510802076s) [noise=true , address=145.79.227.46:4601 , piece_id=GIT43LEIHZQCHTJD6TEUQG5MORR3ACMNVM6DO6KL5MWEL4VAZL4A, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.639771722 (504h7m59.510854757s) [noise=true , address=145.79.227.45:4229 , piece_id=UQRKI64XJWWMSGMF47KT24HQ6H36F3FCW3FNHS2FGRUTUCS7UJTA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640155994 (504h7m59.511239029s) [noise=true , address=145.79.227.16:4909 , piece_id=AXUB5V6YFIPTQSCXABQYGPZCZH3FP635KZX2LIQUKLTA3GV2E4JA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640175110 (504h7m59.511258145s) [noise=true , address=145.79.227.41:4085 , piece_id=SKRC7554EXFUJFEM67JZEGYZNVBCSDNQTEH73NJP3NHGEZBNXCTA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640529695 (504h7m59.51161274s) [noise=true , address=145.79.227.24:4581 , piece_id=GTUIO6HDVB4QB7OCQBV52THCXVLD3XNW265RSECIBFWKCO4FVSRA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640633152 (504h7m59.511716187s) [noise=true , address=45.91.116.12:4225 , piece_id=7C4K6563Y6PEBHIQUR3EG62OERSEMBXCKIKV4GOTXX2JWMIBGVXQ, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.640929887 (504h7m59.512012922s) [noise=true , address=77.90.31.19:4842 , piece_id=25AH5JQSXIJWC2TWRYHUSLFROHXLDVZDIU42EPUIW7IXCRDNMRJA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.641465527 (504h7m59.512548562s) [noise=true , address=77.90.31.21:4878 , piece_id=EC7SD4RCOCTALJFWXCS7IJJGK72SLOVWEHDGX5QH4KVSSTR4WNLQ, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.641564465 (504h7m59.5126475s) [noise=true , address=145.79.227.8:4330 , piece_id=IZJAKOAHRK4PG3DVUVKV5PXUFM4SMLAGGL2K6SZJ3IZ5FV7HBJTA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.641924482 (504h7m59.513007517s) [noise=true , address=45.91.116.13:4154 , piece_id=WL4RPQ4GKZWCUHFMJC3XQODIJ3QS3FKBB5GOV4OKCCCYRQ2LQAGA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.642115385 (504h7m59.51319843s) [noise=true , address=145.79.227.11:4731 , piece_id=ML5US3WRCDFA2NV2VU3BLISZFOZ3V2HCK3KJBGUQNAD5S32627TQ, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.642397251 (504h7m59.513480286s) [noise=true , address=145.79.227.39:4013 , piece_id=DBKSPPPEJAZ2FAESVEZ2MNILJ4XQRG62D3RZIPT7ILFIZ7FDCPIA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:36:59.642883207 (504h7m59.513966242s) [noise=true , address=145.79.227.52:4446 , piece_id=FFPXLJZ4WSIPSTAYCBSIEJ6FQMHATLYHFQTQCJZUA6F7HJICDCLA, seg_pos={0 0}, upload=442389]: Uploading piece... Dec 26 15:37:01.354857134 (504h8m1.225940179s) [upload=442389]: Next returned nil segment Dec 26 15:37:01.354892431 (504h8m1.225975496s) [upload=442389]: Waiting for error group managing segments... Dec 26 15:37:01.649219571 (504h8m1.520302616s) [noise=true , address=145.79.227.52:4446 , piece_id=FFPXLJZ4WSIPSTAYCBSIEJ6FQMHATLYHFQTQCJZUA6F7HJICDCLA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.650584511 (504h8m1.521667566s) [noise=true , address=145.79.227.46:4601 , piece_id=GIT43LEIHZQCHTJD6TEUQG5MORR3ACMNVM6DO6KL5MWEL4VAZL4A, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.654761024 (504h8m1.525844069s) [noise=true , address=145.79.227.45:4229 , piece_id=UQRKI64XJWWMSGMF47KT24HQ6H36F3FCW3FNHS2FGRUTUCS7UJTA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.660002704 (504h8m1.531085739s) [noise=true , address=145.79.227.8:4330 , piece_id=IZJAKOAHRK4PG3DVUVKV5PXUFM4SMLAGGL2K6SZJ3IZ5FV7HBJTA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.666938421 (504h8m1.538021466s) [noise=true , address=145.79.227.16:4909 , piece_id=AXUB5V6YFIPTQSCXABQYGPZCZH3FP635KZX2LIQUKLTA3GV2E4JA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.667973933 (504h8m1.539056978s) [noise=true , address=145.79.227.11:4731 , piece_id=ML5US3WRCDFA2NV2VU3BLISZFOZ3V2HCK3KJBGUQNAD5S32627TQ, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.671117097 (504h8m1.542200173s) [noise=true , address=145.79.227.51:4410 , piece_id=EWWD6SPZ27UWLXYG5XJNM5YGBRJJI3ULDPLGO4GXO64GJUAIQO7A, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.676015876 (504h8m1.547098921s) [noise=true , address=145.79.227.39:4013 , piece_id=DBKSPPPEJAZ2FAESVEZ2MNILJ4XQRG62D3RZIPT7ILFIZ7FDCPIA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.678757887 (504h8m1.549840932s) [noise=true , address=145.79.227.41:4085 , piece_id=SKRC7554EXFUJFEM67JZEGYZNVBCSDNQTEH73NJP3NHGEZBNXCTA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.679445677 (504h8m1.550528732s) [noise=true , address=145.79.227.24:4581 , piece_id=GTUIO6HDVB4QB7OCQBV52THCXVLD3XNW265RSECIBFWKCO4FVSRA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.933941902 (504h8m1.805024947s) [noise=true , address=45.91.116.11:4120 , piece_id=Z6GQDPLUM5D3TBFUNES2E5SVGD5LWCWRFB76BPULF2INUQ4DAMFA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:01.999451834 (504h8m1.87053487s) [noise=true , address=77.90.31.21:4878 , piece_id=EC7SD4RCOCTALJFWXCS7IJJGK72SLOVWEHDGX5QH4KVSSTR4WNLQ, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:02.786685434 (504h8m2.65776848s) [noise=true , address=77.90.31.19:4842 , piece_id=25AH5JQSXIJWC2TWRYHUSLFROHXLDVZDIU42EPUIW7IXCRDNMRJA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: <nil> Dec 26 15:37:02.786708148 (504h8m2.657791183s) [seg_pos={0 0}, upload=442389]: Segment reached optimal threshold of 13 pieces. Dec 26 15:37:02.787036433 (504h8m2.658119478s) [noise=true , address=45.91.116.12:4225 , piece_id=7C4K6563Y6PEBHIQUR3EG62OERSEMBXCKIKV4GOTXX2JWMIBGVXQ, seg_pos={0 0}, upload=442389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 15:37:02.787044568 (504h8m2.658127613s) [noise=true , address=45.91.116.12:4225 , piece_id=7C4K6563Y6PEBHIQUR3EG62OERSEMBXCKIKV4GOTXX2JWMIBGVXQ, seg_pos={0 0}, upload=442389]: Long tail cancellation due to optimal threshold Dec 26 15:37:02.787282050 (504h8m2.658365095s) [noise=true , address=45.91.116.13:4154 , piece_id=WL4RPQ4GKZWCUHFMJC3XQODIJ3QS3FKBB5GOV4OKCCCYRQ2LQAGA, seg_pos={0 0}, upload=442389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 15:37:02.787292561 (504h8m2.658375596s) [noise=true , address=45.91.116.13:4154 , piece_id=WL4RPQ4GKZWCUHFMJC3XQODIJ3QS3FKBB5GOV4OKCCCYRQ2LQAGA, seg_pos={0 0}, upload=442389]: Long tail cancellation due to optimal threshold Dec 26 15:37:02.787347155 (504h8m2.6584302s) [noise=true , address=77.90.31.12:4590 , piece_id=I4YVEPX2FYY5X2XYN2JL5B56XQBD6RPY445OLBS376YOQIXCFQWQ, seg_pos={0 0}, upload=442389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: read tcp 188.166.225.170:38990->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:37:02.787363666 (504h8m2.658446712s) [noise=true , address=77.90.31.12:4590 , piece_id=I4YVEPX2FYY5X2XYN2JL5B56XQBD6RPY445OLBS376YOQIXCFQWQ, seg_pos={0 0}, upload=442389]: Long tail cancellation due to optimal threshold Dec 26 15:37:02.787399845 (504h8m2.65848289s) [seg_pos={0 0}, upload=442389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:37:02.787495687 (504h8m2.658578722s) [upload=442389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:37:02.787500858 (504h8m2.658583903s) [upload=442389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:37:03.086377230 (504h8m2.957460295s) [upload=442389]: Done uploading segments... Dec 26 15:37:03.086403470 (504h8m2.957486516s) [upload=442389]: Upload finished. err: <nil> Duration: 30.003595372s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX] duration: 5.001355824s trace: https://jaeger.dev.storj.io/trace/6455ce761026ebfd FAIL (2025-12-26 15:37:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001382725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...X] duration: 5.001608706s trace: https://jaeger.dev.storj.io/trace/7b5d0425283e8d05 FAIL (2025-12-26 15:37:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00163711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX..X.X.....X.....X.X............XXX....XX...X......X..X..] duration: 6.092213168s trace: https://jaeger.dev.storj.io/trace/9be7c0c11f39bee FAIL (2025-12-26 15:36:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad Dec 26 15:34:24.826087459 (504h5m24.697170494s) [upload=442354]: Done starting upload Dec 26 15:34:24.826260930 (504h5m24.697343975s) [upload=442354]: Uploading segments... Dec 26 15:34:24.826352635 (504h5m24.69743568s) [upload=442354]: Got next segment. Inline: false Dec 26 15:34:24.826393142 (504h5m24.697476177s) [upload=442354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:34:24.826398652 (504h5m24.697481697s) [upload=442354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:34:25.069136767 (504h5m24.940219802s) [seg_pos={0 0}, upload=442354]: Begin upload segment... Dec 26 15:34:25.069197152 (504h5m24.940280197s) [seg_pos={0 0}, upload=442354]: Done begin upload segment. Dec 26 15:34:25.069206269 (504h5m24.940289305s) [noise=true , address=145.79.227.49:4338 , piece_id=OMPXSOKMSP3R5IS37BSOLQEYS7CIAU5JPXNLIYUCSA2CJNL2AA5Q, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.069280942 (504h5m24.940363978s) [noise=true , address=145.79.227.12:4767 , piece_id=AUAHK44K72XSUM4TAFLU3OI3XRPDR6VPXXO3MGGA2NJHNZENMQIA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.069683138 (504h5m24.940766173s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.070066848 (504h5m24.941149883s) [noise=true , address=145.79.227.42:4121 , piece_id=SA2SSBVL4UJTKT73Y4DOYMF2Q2M7VLZPGYOHFFQZAOPC4SKTXCYQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.070864417 (504h5m24.941947452s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071153237 (504h5m24.942236272s) [noise=true , address=45.91.116.9:4262 , piece_id=JRJSX37GU6WCQRWQZ7ZMKJR547LZ62R7YBPZSD5H22DNU3YFOUMA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071418823 (504h5m24.942501858s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071696111 (504h5m24.942779156s) [noise=true , address=145.79.227.37:4566 , piece_id=NMFBRVNHEBDBBZG6DDQEOOWPF22R3IEZU37RE6NNWWF6ZRR4AR5A, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.072014076 (504h5m24.943097121s) [noise=true , address=45.91.116.13:4153 , piece_id=S3ZQ6GQCMDW2RXDFQ3WUVS2HKAYVJR322NB6QVU46TYEWPJOJENQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.072901637 (504h5m24.943984672s) [noise=true , address=45.91.116.14:4189 , piece_id=VZHRPJDDV23UFDULZJK6LMB6K7KVGS6NELYBXYFYANERN4FUMRCQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.073032866 (504h5m24.944115911s) [noise=true , address=145.79.227.21:4475 , piece_id=IQFEFERWXBGHEHTYDQJZBVSMQDNSC62ZMIGSNFGJIDH3ADJ4QMYQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.074164171 (504h5m24.945247216s) [noise=true , address=145.79.227.7:4294 , piece_id=R3FNMICJODJ6BRU25W56S7FMW5NRHBEFQ4HQCLGOGPPGG4Y6FYQA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.074528744 (504h5m24.94561178s) [noise=true , address=145.79.227.2:4114 , piece_id=J4VHYAVXBOYT2FJFVHMT77L5IRTJD62JHMTB6UPIPECQ3C3FNAMQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.075153324 (504h5m24.946236369s) [noise=true , address=145.79.227.52:4446 , piece_id=T32XKE7LNKTCVH5W6CQDHH57AG75VPB24ANBJIURHXEAMVZTKYZQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.076188485 (504h5m24.94727152s) [noise=true , address=145.79.227.16:4910 , piece_id=VDB3UWC7KP66HITDIHUPHLE6Y4Q26I4HS3WRVTJRTJLRIE2L7U5Q, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.077403609 (504h5m24.948486644s) [noise=true , address=145.79.227.24:4581 , piece_id=MMXOFNJLE2GEGENCKB5PCXJD3OXPOTJYCZJAGPFCEMXPJHNWNOKQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:26.795877152 (504h5m26.666960197s) [upload=442354]: Next returned nil segment Dec 26 15:34:26.795898002 (504h5m26.666981047s) [upload=442354]: Waiting for error group managing segments... Dec 26 15:34:27.105765855 (504h5m26.97684889s) [noise=true , address=145.79.227.2:4114 , piece_id=J4VHYAVXBOYT2FJFVHMT77L5IRTJD62JHMTB6UPIPECQ3C3FNAMQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.108123354 (504h5m26.979206399s) [noise=true , address=145.79.227.16:4910 , piece_id=VDB3UWC7KP66HITDIHUPHLE6Y4Q26I4HS3WRVTJRTJLRIE2L7U5Q, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.109456793 (504h5m26.980539828s) [noise=true , address=145.79.227.12:4767 , piece_id=AUAHK44K72XSUM4TAFLU3OI3XRPDR6VPXXO3MGGA2NJHNZENMQIA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112117218 (504h5m26.983200253s) [noise=true , address=145.79.227.24:4581 , piece_id=MMXOFNJLE2GEGENCKB5PCXJD3OXPOTJYCZJAGPFCEMXPJHNWNOKQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112432809 (504h5m26.983515844s) [noise=true , address=145.79.227.37:4566 , piece_id=NMFBRVNHEBDBBZG6DDQEOOWPF22R3IEZU37RE6NNWWF6ZRR4AR5A, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112853281 (504h5m26.983936316s) [noise=true , address=145.79.227.52:4446 , piece_id=T32XKE7LNKTCVH5W6CQDHH57AG75VPB24ANBJIURHXEAMVZTKYZQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.128952075 (504h5m27.00003511s) [noise=true , address=145.79.227.49:4338 , piece_id=OMPXSOKMSP3R5IS37BSOLQEYS7CIAU5JPXNLIYUCSA2CJNL2AA5Q, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.131756325 (504h5m27.00283936s) [noise=true , address=145.79.227.7:4294 , piece_id=R3FNMICJODJ6BRU25W56S7FMW5NRHBEFQ4HQCLGOGPPGG4Y6FYQA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.133890639 (504h5m27.004973684s) [noise=true , address=145.79.227.21:4475 , piece_id=IQFEFERWXBGHEHTYDQJZBVSMQDNSC62ZMIGSNFGJIDH3ADJ4QMYQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.138440283 (504h5m27.009523318s) [noise=true , address=145.79.227.42:4121 , piece_id=SA2SSBVL4UJTKT73Y4DOYMF2Q2M7VLZPGYOHFFQZAOPC4SKTXCYQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.397490890 (504h5m27.268573935s) [noise=true , address=45.91.116.14:4189 , piece_id=VZHRPJDDV23UFDULZJK6LMB6K7KVGS6NELYBXYFYANERN4FUMRCQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.459581161 (504h5m27.330664206s) [noise=true , address=45.91.116.13:4153 , piece_id=S3ZQ6GQCMDW2RXDFQ3WUVS2HKAYVJR322NB6QVU46TYEWPJOJENQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.729226993 (504h5m27.600310038s) [noise=true , address=45.91.116.9:4262 , piece_id=JRJSX37GU6WCQRWQZ7ZMKJR547LZ62R7YBPZSD5H22DNU3YFOUMA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.729249386 (504h5m27.600332421s) [seg_pos={0 0}, upload=442354]: Segment reached optimal threshold of 13 pieces. Dec 26 15:34:27.729537935 (504h5m27.600620981s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection Dec 26 15:34:27.729547844 (504h5m27.600630899s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.729589394 (504h5m27.600672439s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 15:34:27.729597139 (504h5m27.600680174s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.729978565 (504h5m27.60106162s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50458->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50458->45.91.116.11:4119: use of closed network connection Dec 26 15:34:27.730003902 (504h5m27.601086947s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.730035853 (504h5m27.601118898s) [seg_pos={0 0}, upload=442354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:34:27.730130644 (504h5m27.601213689s) [upload=442354]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:34:27.730136014 (504h5m27.601219049s) [upload=442354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:34:28.030174941 (504h5m27.901257976s) [upload=442354]: Done uploading segments... Dec 26 15:34:28.030196833 (504h5m27.901279868s) [upload=442354]: Upload finished. err: <nil> Duration: 6.092239818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X..X..X.X...XX.XXXX.X..................X..........X.XX.X] duration: 1m0.002436642s trace: https://jaeger.dev.storj.io/trace/18fbdbd65f37dab6 FAIL (2025-12-26 15:36:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ) context deadline exceeded; piecestore: (Node ID: 12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, Piece ID: IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: 5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA) context deadline exceeded ======LOGS====== 22141s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119456000 (504h6m34.990539035s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.119767683 (504h6m34.990850728s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120122229 (504h6m34.991205264s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120490921 (504h6m34.991573956s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.120909187 (504h6m34.991992222s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121373372 (504h6m34.992456407s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121393289 (504h6m34.992476334s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.121900044 (504h6m34.992983089s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122331596 (504h6m34.993414631s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.122778266 (504h6m34.993861311s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123296363 (504h6m34.994379408s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.123820291 (504h6m34.994903336s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124272331 (504h6m34.995355376s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:35.124440031 (504h6m34.995523066s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Uploading piece... Dec 26 15:35:36.442690439 (504h6m36.313773484s) [upload=442372]: Next returned nil segment Dec 26 15:35:36.442714626 (504h6m36.313797661s) [upload=442372]: Waiting for error group managing segments... Dec 26 15:35:36.799517290 (504h6m36.670600325s) [noise=true , address=64.49.13.140:4508 , piece_id=XJ5WNDMGQOKYEDOARY2E6YCFX2BXT4JO5OY6K36GXEIIEQKGDDGQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.164491694 (504h6m37.035574739s) [noise=true , address=77.90.31.12:4616 , piece_id=VOZHEBCPD64RT37OGWS2SL6J6SAYOUOKU5PE2UGUGOWPERAGWORQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.169500051 (504h6m37.040583096s) [noise=true , address=77.90.31.15:4714 , piece_id=VCQB7TZYN47R4ITUJWCUODALE3NR6YDQIIHBU3WQZCNSWX4G2Q5A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.222379363 (504h6m37.093462398s) [noise=true , address=45.91.116.5:4317 , piece_id=HT6QNVNEZQVUV5M2NIBMHG6UIQCJMYMFBB3EC7VTMDF3K4PLWATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.230139129 (504h6m37.101222174s) [noise=true , address=45.91.116.4:4962 , piece_id=6MDBE4OAV3J6F5A4KQDODIEPKCKKRJQEQ6PLU5WSN2TQOQLBNQEA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.335458906 (504h6m37.206541951s) [noise=true , address=77.90.31.14:4680 , piece_id=IL5HEBCHDXXEM2NB6LZPZTMLXWIF6IVB364NTMLTUGFVADVN5Z5Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.368208557 (504h6m37.239291592s) [noise=true , address=77.90.31.16:4757 , piece_id=KTP5KVZPWXG4PSX6DBXGZHWLAKCJYSXBF5OXOQC37DWISNIYGATQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.428078270 (504h6m37.299161305s) [noise=true , address=45.91.116.10:4097 , piece_id=JADDCS26PB6CR2BY64BHPUL2RMN7H3QR4L4CVIU4QZC736UFY4XQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.513975203 (504h6m37.385058238s) [noise=true , address=77.90.31.21:4896 , piece_id=KGWHXUVJU4347A5Z5OXPLTKH3XKW3L5TYMZ3IULIWZKHLNSENI7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.835023545 (504h6m37.70610659s) [noise=true , address=145.79.227.34:4928 , piece_id=KNGLYETZJTW2FPYFBOJYNR7KWOC6RUMQHIHCU76XO3QWT73HR7DQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.908251443 (504h6m37.779334489s) [noise=true , address=145.79.227.44:4199 , piece_id=EBKJMEKSVZFENTMMFMOXLNULFRBR6NPPISVZFTZDPAZBBMPLDFIQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.944654971 (504h6m37.815738016s) [noise=true , address=145.79.227.30:4769 , piece_id=UEPN5QSXPKQUJBWJJMHSNZXFEZGXWQ5JYZBHTOT5IK7F2NZSJ3UQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:37.996043885 (504h6m37.867126931s) [noise=true , address=145.79.227.40:4065 , piece_id=FGW4R2XGE54RBBOBRSOJ4BSUZ5VFL772BWYLTNYG4JSDEXG75TKQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217717732 (504h6m38.088800777s) [noise=true , address=45.91.116.14:4220 , piece_id=5BIOKP2DS3LNBPCEZ42OAN4UCLE3J3K7ZAYVNVTVJW3QCATG3BFA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: <nil> Dec 26 15:35:38.217742618 (504h6m38.088825653s) [seg_pos={0 0}, upload=442372]: Segment reached optimal threshold of 14 pieces. Dec 26 15:35:38.218057368 (504h6m38.089140413s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218068630 (504h6m38.089151675s) [noise=true , address=145.79.227.43:4178 , piece_id=JUOU5RMF34JAS7I42QQKVFNTILZTWM62CPJ3DVMHPM4G4C2OKEUA, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218083247 (504h6m38.089166282s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39174->77.90.31.11:4574: use of closed network connection Dec 26 15:35:38.218099709 (504h6m38.089182744s) [noise=true , address=77.90.31.11:4574 , piece_id=UBVWY4VHF5FZDOR4KZUC4WYHNHHP44CD3MRQNJYWYDKVOJSKNTWQ, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218246418 (504h6m38.089329453s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 26 15:35:38.218256748 (504h6m38.089339783s) [noise=true , address=77.90.31.19:4855 , piece_id=ZMPGXVNR4KO5GBSSX5DUEM2PEP4S3F3ENQSBZ4NG2D3AT3HAQM7Q, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218418897 (504h6m38.089501932s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58694->64.49.13.169:4705: use of closed network connection Dec 26 15:35:38.218427152 (504h6m38.089510187s) [noise=true , address=64.49.13.169:4705 , piece_id=66E6Z7VLHH6JDZ3E3MVF3P4WNAHSLF57W4L75INVUKGO3PLSYP3A, seg_pos={0 0}, upload=442372]: Long tail cancellation due to optimal threshold Dec 26 15:35:38.218594601 (504h6m38.089677636s) [seg_pos={0 0}, upload=442372]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:35:38.218708177 (504h6m38.089791212s) [upload=442372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:35:38.218718356 (504h6m38.089801391s) [upload=442372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:35:38.513584834 (504h6m38.38466787s) [upload=442372]: Done uploading segments... Dec 26 15:35:38.513609711 (504h6m38.384692756s) [upload=442372]: Upload finished. err: <nil> Duration: 1m0.002447802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X...] duration: 3.38911715s trace: https://jaeger.dev.storj.io/trace/71972eeda6034f51 FAIL (2025-12-26 15:36:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.389148891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X..] duration: 5.001517151s trace: https://jaeger.dev.storj.io/trace/71972eeda6034f51 FAIL (2025-12-26 15:35:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001534143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...XX..X.X.....X.....X.X............XXX....XX...X......X..X.] duration: 5.789281372s trace: https://jaeger.dev.storj.io/trace/9be7c0c11f39bee FAIL (2025-12-26 15:35:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad Dec 26 15:34:24.826087459 (504h5m24.697170494s) [upload=442354]: Done starting upload Dec 26 15:34:24.826260930 (504h5m24.697343975s) [upload=442354]: Uploading segments... Dec 26 15:34:24.826352635 (504h5m24.69743568s) [upload=442354]: Got next segment. Inline: false Dec 26 15:34:24.826393142 (504h5m24.697476177s) [upload=442354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:34:24.826398652 (504h5m24.697481697s) [upload=442354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:34:25.069136767 (504h5m24.940219802s) [seg_pos={0 0}, upload=442354]: Begin upload segment... Dec 26 15:34:25.069197152 (504h5m24.940280197s) [seg_pos={0 0}, upload=442354]: Done begin upload segment. Dec 26 15:34:25.069206269 (504h5m24.940289305s) [noise=true , address=145.79.227.49:4338 , piece_id=OMPXSOKMSP3R5IS37BSOLQEYS7CIAU5JPXNLIYUCSA2CJNL2AA5Q, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.069280942 (504h5m24.940363978s) [noise=true , address=145.79.227.12:4767 , piece_id=AUAHK44K72XSUM4TAFLU3OI3XRPDR6VPXXO3MGGA2NJHNZENMQIA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.069683138 (504h5m24.940766173s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.070066848 (504h5m24.941149883s) [noise=true , address=145.79.227.42:4121 , piece_id=SA2SSBVL4UJTKT73Y4DOYMF2Q2M7VLZPGYOHFFQZAOPC4SKTXCYQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.070864417 (504h5m24.941947452s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071153237 (504h5m24.942236272s) [noise=true , address=45.91.116.9:4262 , piece_id=JRJSX37GU6WCQRWQZ7ZMKJR547LZ62R7YBPZSD5H22DNU3YFOUMA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071418823 (504h5m24.942501858s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071696111 (504h5m24.942779156s) [noise=true , address=145.79.227.37:4566 , piece_id=NMFBRVNHEBDBBZG6DDQEOOWPF22R3IEZU37RE6NNWWF6ZRR4AR5A, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.072014076 (504h5m24.943097121s) [noise=true , address=45.91.116.13:4153 , piece_id=S3ZQ6GQCMDW2RXDFQ3WUVS2HKAYVJR322NB6QVU46TYEWPJOJENQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.072901637 (504h5m24.943984672s) [noise=true , address=45.91.116.14:4189 , piece_id=VZHRPJDDV23UFDULZJK6LMB6K7KVGS6NELYBXYFYANERN4FUMRCQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.073032866 (504h5m24.944115911s) [noise=true , address=145.79.227.21:4475 , piece_id=IQFEFERWXBGHEHTYDQJZBVSMQDNSC62ZMIGSNFGJIDH3ADJ4QMYQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.074164171 (504h5m24.945247216s) [noise=true , address=145.79.227.7:4294 , piece_id=R3FNMICJODJ6BRU25W56S7FMW5NRHBEFQ4HQCLGOGPPGG4Y6FYQA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.074528744 (504h5m24.94561178s) [noise=true , address=145.79.227.2:4114 , piece_id=J4VHYAVXBOYT2FJFVHMT77L5IRTJD62JHMTB6UPIPECQ3C3FNAMQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.075153324 (504h5m24.946236369s) [noise=true , address=145.79.227.52:4446 , piece_id=T32XKE7LNKTCVH5W6CQDHH57AG75VPB24ANBJIURHXEAMVZTKYZQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.076188485 (504h5m24.94727152s) [noise=true , address=145.79.227.16:4910 , piece_id=VDB3UWC7KP66HITDIHUPHLE6Y4Q26I4HS3WRVTJRTJLRIE2L7U5Q, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.077403609 (504h5m24.948486644s) [noise=true , address=145.79.227.24:4581 , piece_id=MMXOFNJLE2GEGENCKB5PCXJD3OXPOTJYCZJAGPFCEMXPJHNWNOKQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:26.795877152 (504h5m26.666960197s) [upload=442354]: Next returned nil segment Dec 26 15:34:26.795898002 (504h5m26.666981047s) [upload=442354]: Waiting for error group managing segments... Dec 26 15:34:27.105765855 (504h5m26.97684889s) [noise=true , address=145.79.227.2:4114 , piece_id=J4VHYAVXBOYT2FJFVHMT77L5IRTJD62JHMTB6UPIPECQ3C3FNAMQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.108123354 (504h5m26.979206399s) [noise=true , address=145.79.227.16:4910 , piece_id=VDB3UWC7KP66HITDIHUPHLE6Y4Q26I4HS3WRVTJRTJLRIE2L7U5Q, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.109456793 (504h5m26.980539828s) [noise=true , address=145.79.227.12:4767 , piece_id=AUAHK44K72XSUM4TAFLU3OI3XRPDR6VPXXO3MGGA2NJHNZENMQIA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112117218 (504h5m26.983200253s) [noise=true , address=145.79.227.24:4581 , piece_id=MMXOFNJLE2GEGENCKB5PCXJD3OXPOTJYCZJAGPFCEMXPJHNWNOKQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112432809 (504h5m26.983515844s) [noise=true , address=145.79.227.37:4566 , piece_id=NMFBRVNHEBDBBZG6DDQEOOWPF22R3IEZU37RE6NNWWF6ZRR4AR5A, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112853281 (504h5m26.983936316s) [noise=true , address=145.79.227.52:4446 , piece_id=T32XKE7LNKTCVH5W6CQDHH57AG75VPB24ANBJIURHXEAMVZTKYZQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.128952075 (504h5m27.00003511s) [noise=true , address=145.79.227.49:4338 , piece_id=OMPXSOKMSP3R5IS37BSOLQEYS7CIAU5JPXNLIYUCSA2CJNL2AA5Q, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.131756325 (504h5m27.00283936s) [noise=true , address=145.79.227.7:4294 , piece_id=R3FNMICJODJ6BRU25W56S7FMW5NRHBEFQ4HQCLGOGPPGG4Y6FYQA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.133890639 (504h5m27.004973684s) [noise=true , address=145.79.227.21:4475 , piece_id=IQFEFERWXBGHEHTYDQJZBVSMQDNSC62ZMIGSNFGJIDH3ADJ4QMYQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.138440283 (504h5m27.009523318s) [noise=true , address=145.79.227.42:4121 , piece_id=SA2SSBVL4UJTKT73Y4DOYMF2Q2M7VLZPGYOHFFQZAOPC4SKTXCYQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.397490890 (504h5m27.268573935s) [noise=true , address=45.91.116.14:4189 , piece_id=VZHRPJDDV23UFDULZJK6LMB6K7KVGS6NELYBXYFYANERN4FUMRCQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.459581161 (504h5m27.330664206s) [noise=true , address=45.91.116.13:4153 , piece_id=S3ZQ6GQCMDW2RXDFQ3WUVS2HKAYVJR322NB6QVU46TYEWPJOJENQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.729226993 (504h5m27.600310038s) [noise=true , address=45.91.116.9:4262 , piece_id=JRJSX37GU6WCQRWQZ7ZMKJR547LZ62R7YBPZSD5H22DNU3YFOUMA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.729249386 (504h5m27.600332421s) [seg_pos={0 0}, upload=442354]: Segment reached optimal threshold of 13 pieces. Dec 26 15:34:27.729537935 (504h5m27.600620981s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection Dec 26 15:34:27.729547844 (504h5m27.600630899s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.729589394 (504h5m27.600672439s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 15:34:27.729597139 (504h5m27.600680174s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.729978565 (504h5m27.60106162s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50458->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50458->45.91.116.11:4119: use of closed network connection Dec 26 15:34:27.730003902 (504h5m27.601086947s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.730035853 (504h5m27.601118898s) [seg_pos={0 0}, upload=442354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:34:27.730130644 (504h5m27.601213689s) [upload=442354]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:34:27.730136014 (504h5m27.601219049s) [upload=442354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:34:28.030174941 (504h5m27.901257976s) [upload=442354]: Done uploading segments... Dec 26 15:34:28.030196833 (504h5m27.901279868s) [upload=442354]: Upload finished. err: <nil> Duration: 5.789308974s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X.X..X..X.X...XX.XXXX.X..................X..........X.XX.] duration: 50.159949617s trace: https://jaeger.dev.storj.io/trace/4d6ec53c41a9b5cc FAIL (2025-12-26 15:35:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, Piece ID: 54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA) context deadline exceeded; piecestore: (Node ID: 1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, Piece ID: 5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ) context deadline exceeded; piecestore: (Node ID: 1CMsUV4eQnyd4FLQYLi5xdPDurkVaL2kyn8EW4VigZsh8sdTv6, Piece ID: VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA) context deadline exceeded ======LOGS====== ODZ2M5DFJDIPA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.484026986 (504h4m44.355110031s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.484477945 (504h4m44.35556099s) [noise=true , address=77.90.31.19:4866 , piece_id=F32RITTYFC6EO5E7NTOTMNLJ2VMEHN46YOXPCWVG25VDPIP4B75Q, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485078588 (504h4m44.356161633s) [noise=true , address=45.91.116.4:4963 , piece_id=5LDE6TJMLXDAFPDJ7ZMKKTS6LS22VDZKW7MWMVEGA2DU37F2PDYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485615390 (504h4m44.356698435s) [noise=true , address=45.91.116.11:4121 , piece_id=5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485637873 (504h4m44.356720918s) [noise=true , address=145.79.227.44:4205 , piece_id=I5ISL5NWNUMQKLO75E7EJTHKN3R2NDZ5LDBH72DMO2PTQS67R2VA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486193952 (504h4m44.357276987s) [noise=true , address=77.90.31.14:4670 , piece_id=DRBNQDD6O3CHX5OYAMYMKS7SRI5A4QOFVOAEKI3GNF3JWUKC55JA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486453666 (504h4m44.357536701s) [noise=true , address=45.91.116.14:4201 , piece_id=IHFNLZRXMP3GFZP65PZSWG5GZ2ZFKTG2YVYC6GQ2BFEGQHS33HBA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486754038 (504h4m44.357837083s) [noise=true , address=64.49.13.170:4725 , piece_id=3XFNXHSV6HMV5IDZWXZ5XH7RWRYS2UXATNGZWQNTVY57DBB23GXQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487114945 (504h4m44.35819799s) [noise=true , address=145.79.227.29:4954 , piece_id=A6GARORLQWVZCU2GTFJ555RIXW7PCZB4I3CUJ6OXXCFIBJCJHBYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487331979 (504h4m44.358415024s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487769442 (504h4m44.358852487s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487874922 (504h4m44.358957967s) [noise=true , address=77.90.31.15:4703 , piece_id=54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.488454076 (504h4m44.359537111s) [noise=true , address=45.91.116.13:4174 , piece_id=VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.491854641 (504h4m44.362937686s) [noise=true , address=64.49.13.26:4945 , piece_id=7ISUES6U4NH3NZU3GUOR2IMXLVRTSPUSWUZNC2QICKZ3MJELN4HA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:45.797674129 (504h4m45.668757174s) [upload=442344]: Next returned nil segment Dec 26 15:33:45.797692585 (504h4m45.66877562s) [upload=442344]: Waiting for error group managing segments... Dec 26 15:33:46.130577946 (504h4m46.001660981s) [noise=true , address=193.149.164.143:4702 , piece_id=ORKNZ2J4IE7SSNJGTYPD5V4VKDJGGKXVE7Z7BW6ODZ2M5DFJDIPA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.205726252 (504h4m46.076809297s) [noise=true , address=64.49.13.26:4945 , piece_id=7ISUES6U4NH3NZU3GUOR2IMXLVRTSPUSWUZNC2QICKZ3MJELN4HA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.308550998 (504h4m46.179634043s) [noise=true , address=64.49.13.170:4725 , piece_id=3XFNXHSV6HMV5IDZWXZ5XH7RWRYS2UXATNGZWQNTVY57DBB23GXQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.577575107 (504h4m46.448658152s) [noise=true , address=45.91.116.14:4201 , piece_id=IHFNLZRXMP3GFZP65PZSWG5GZ2ZFKTG2YVYC6GQ2BFEGQHS33HBA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.615742895 (504h4m46.48682593s) [noise=true , address=45.91.116.11:4121 , piece_id=5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.616326656 (504h4m46.487409691s) [noise=true , address=45.91.116.13:4174 , piece_id=VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.699340129 (504h4m46.570423174s) [noise=true , address=45.91.116.4:4963 , piece_id=5LDE6TJMLXDAFPDJ7ZMKKTS6LS22VDZKW7MWMVEGA2DU37F2PDYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.708773672 (504h4m46.579856707s) [noise=true , address=145.79.227.29:4954 , piece_id=A6GARORLQWVZCU2GTFJ555RIXW7PCZB4I3CUJ6OXXCFIBJCJHBYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.955154153 (504h4m46.826237198s) [noise=true , address=77.90.31.15:4703 , piece_id=54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.142693852 (504h4m47.013776897s) [noise=true , address=77.90.31.19:4866 , piece_id=F32RITTYFC6EO5E7NTOTMNLJ2VMEHN46YOXPCWVG25VDPIP4B75Q, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.276150466 (504h4m47.147233511s) [noise=true , address=145.79.227.40:4053 , piece_id=2J2EY3X3WFXWWXZBHC6R2EN3SSOT35BJJULBJWLT577PJA6DY7CQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.304118842 (504h4m47.175201877s) [noise=true , address=145.79.227.44:4205 , piece_id=I5ISL5NWNUMQKLO75E7EJTHKN3R2NDZ5LDBH72DMO2PTQS67R2VA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.440569149 (504h4m47.311652194s) [noise=true , address=145.79.227.30:4796 , piece_id=MYHNLDAKFLPH2A7JEVQCAT7YQAO54XJ2SFD2MGQPPAFVIDVO47KA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.449458225 (504h4m47.32054128s) [noise=true , address=77.90.31.14:4670 , piece_id=DRBNQDD6O3CHX5OYAMYMKS7SRI5A4QOFVOAEKI3GNF3JWUKC55JA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.449479796 (504h4m47.320562841s) [seg_pos={0 0}, upload=442344]: Segment reached optimal threshold of 14 pieces. Dec 26 15:33:47.449793704 (504h4m47.320876739s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection; write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection Dec 26 15:33:47.449856813 (504h4m47.320939858s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.449994956 (504h4m47.321078001s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection Dec 26 15:33:47.450015105 (504h4m47.32109815s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450029161 (504h4m47.321112206s) [noise=true , address=77.90.31.11:4576 , piece_id=T5UX5RSTPFYOI6OS744JCEB4NNNBLYP24XCW7LOEYTDPTKHXAGAQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv): protocol: read tcp 188.166.225.170:37370->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:33:47.450036004 (504h4m47.321119039s) [noise=true , address=77.90.31.11:4576 , piece_id=T5UX5RSTPFYOI6OS744JCEB4NNNBLYP24XCW7LOEYTDPTKHXAGAQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450179719 (504h4m47.321262764s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 15:33:47.450190198 (504h4m47.321273243s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450218202 (504h4m47.321301247s) [seg_pos={0 0}, upload=442344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:33:47.450321839 (504h4m47.321404884s) [upload=442344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:33:47.450328582 (504h4m47.321411637s) [upload=442344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:33:47.743361678 (504h4m47.614444733s) [upload=442344]: Done uploading segments... Dec 26 15:33:47.743387397 (504h4m47.614470442s) [upload=442344]: Upload finished. err: <nil> Duration: 50.159962974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX..X.X.....X.....X.X............XXX....XX...X......X..X] duration: 30.003059451s trace: https://jaeger.dev.storj.io/trace/9be7c0c11f39bee FAIL (2025-12-26 15:34:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad Dec 26 15:34:24.826087459 (504h5m24.697170494s) [upload=442354]: Done starting upload Dec 26 15:34:24.826260930 (504h5m24.697343975s) [upload=442354]: Uploading segments... Dec 26 15:34:24.826352635 (504h5m24.69743568s) [upload=442354]: Got next segment. Inline: false Dec 26 15:34:24.826393142 (504h5m24.697476177s) [upload=442354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:34:24.826398652 (504h5m24.697481697s) [upload=442354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:34:25.069136767 (504h5m24.940219802s) [seg_pos={0 0}, upload=442354]: Begin upload segment... Dec 26 15:34:25.069197152 (504h5m24.940280197s) [seg_pos={0 0}, upload=442354]: Done begin upload segment. Dec 26 15:34:25.069206269 (504h5m24.940289305s) [noise=true , address=145.79.227.49:4338 , piece_id=OMPXSOKMSP3R5IS37BSOLQEYS7CIAU5JPXNLIYUCSA2CJNL2AA5Q, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.069280942 (504h5m24.940363978s) [noise=true , address=145.79.227.12:4767 , piece_id=AUAHK44K72XSUM4TAFLU3OI3XRPDR6VPXXO3MGGA2NJHNZENMQIA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.069683138 (504h5m24.940766173s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.070066848 (504h5m24.941149883s) [noise=true , address=145.79.227.42:4121 , piece_id=SA2SSBVL4UJTKT73Y4DOYMF2Q2M7VLZPGYOHFFQZAOPC4SKTXCYQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.070864417 (504h5m24.941947452s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071153237 (504h5m24.942236272s) [noise=true , address=45.91.116.9:4262 , piece_id=JRJSX37GU6WCQRWQZ7ZMKJR547LZ62R7YBPZSD5H22DNU3YFOUMA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071418823 (504h5m24.942501858s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071696111 (504h5m24.942779156s) [noise=true , address=145.79.227.37:4566 , piece_id=NMFBRVNHEBDBBZG6DDQEOOWPF22R3IEZU37RE6NNWWF6ZRR4AR5A, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.072014076 (504h5m24.943097121s) [noise=true , address=45.91.116.13:4153 , piece_id=S3ZQ6GQCMDW2RXDFQ3WUVS2HKAYVJR322NB6QVU46TYEWPJOJENQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.072901637 (504h5m24.943984672s) [noise=true , address=45.91.116.14:4189 , piece_id=VZHRPJDDV23UFDULZJK6LMB6K7KVGS6NELYBXYFYANERN4FUMRCQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.073032866 (504h5m24.944115911s) [noise=true , address=145.79.227.21:4475 , piece_id=IQFEFERWXBGHEHTYDQJZBVSMQDNSC62ZMIGSNFGJIDH3ADJ4QMYQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.074164171 (504h5m24.945247216s) [noise=true , address=145.79.227.7:4294 , piece_id=R3FNMICJODJ6BRU25W56S7FMW5NRHBEFQ4HQCLGOGPPGG4Y6FYQA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.074528744 (504h5m24.94561178s) [noise=true , address=145.79.227.2:4114 , piece_id=J4VHYAVXBOYT2FJFVHMT77L5IRTJD62JHMTB6UPIPECQ3C3FNAMQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.075153324 (504h5m24.946236369s) [noise=true , address=145.79.227.52:4446 , piece_id=T32XKE7LNKTCVH5W6CQDHH57AG75VPB24ANBJIURHXEAMVZTKYZQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.076188485 (504h5m24.94727152s) [noise=true , address=145.79.227.16:4910 , piece_id=VDB3UWC7KP66HITDIHUPHLE6Y4Q26I4HS3WRVTJRTJLRIE2L7U5Q, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.077403609 (504h5m24.948486644s) [noise=true , address=145.79.227.24:4581 , piece_id=MMXOFNJLE2GEGENCKB5PCXJD3OXPOTJYCZJAGPFCEMXPJHNWNOKQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:26.795877152 (504h5m26.666960197s) [upload=442354]: Next returned nil segment Dec 26 15:34:26.795898002 (504h5m26.666981047s) [upload=442354]: Waiting for error group managing segments... Dec 26 15:34:27.105765855 (504h5m26.97684889s) [noise=true , address=145.79.227.2:4114 , piece_id=J4VHYAVXBOYT2FJFVHMT77L5IRTJD62JHMTB6UPIPECQ3C3FNAMQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.108123354 (504h5m26.979206399s) [noise=true , address=145.79.227.16:4910 , piece_id=VDB3UWC7KP66HITDIHUPHLE6Y4Q26I4HS3WRVTJRTJLRIE2L7U5Q, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.109456793 (504h5m26.980539828s) [noise=true , address=145.79.227.12:4767 , piece_id=AUAHK44K72XSUM4TAFLU3OI3XRPDR6VPXXO3MGGA2NJHNZENMQIA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112117218 (504h5m26.983200253s) [noise=true , address=145.79.227.24:4581 , piece_id=MMXOFNJLE2GEGENCKB5PCXJD3OXPOTJYCZJAGPFCEMXPJHNWNOKQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112432809 (504h5m26.983515844s) [noise=true , address=145.79.227.37:4566 , piece_id=NMFBRVNHEBDBBZG6DDQEOOWPF22R3IEZU37RE6NNWWF6ZRR4AR5A, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112853281 (504h5m26.983936316s) [noise=true , address=145.79.227.52:4446 , piece_id=T32XKE7LNKTCVH5W6CQDHH57AG75VPB24ANBJIURHXEAMVZTKYZQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.128952075 (504h5m27.00003511s) [noise=true , address=145.79.227.49:4338 , piece_id=OMPXSOKMSP3R5IS37BSOLQEYS7CIAU5JPXNLIYUCSA2CJNL2AA5Q, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.131756325 (504h5m27.00283936s) [noise=true , address=145.79.227.7:4294 , piece_id=R3FNMICJODJ6BRU25W56S7FMW5NRHBEFQ4HQCLGOGPPGG4Y6FYQA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.133890639 (504h5m27.004973684s) [noise=true , address=145.79.227.21:4475 , piece_id=IQFEFERWXBGHEHTYDQJZBVSMQDNSC62ZMIGSNFGJIDH3ADJ4QMYQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.138440283 (504h5m27.009523318s) [noise=true , address=145.79.227.42:4121 , piece_id=SA2SSBVL4UJTKT73Y4DOYMF2Q2M7VLZPGYOHFFQZAOPC4SKTXCYQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.397490890 (504h5m27.268573935s) [noise=true , address=45.91.116.14:4189 , piece_id=VZHRPJDDV23UFDULZJK6LMB6K7KVGS6NELYBXYFYANERN4FUMRCQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.459581161 (504h5m27.330664206s) [noise=true , address=45.91.116.13:4153 , piece_id=S3ZQ6GQCMDW2RXDFQ3WUVS2HKAYVJR322NB6QVU46TYEWPJOJENQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.729226993 (504h5m27.600310038s) [noise=true , address=45.91.116.9:4262 , piece_id=JRJSX37GU6WCQRWQZ7ZMKJR547LZ62R7YBPZSD5H22DNU3YFOUMA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.729249386 (504h5m27.600332421s) [seg_pos={0 0}, upload=442354]: Segment reached optimal threshold of 13 pieces. Dec 26 15:34:27.729537935 (504h5m27.600620981s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection Dec 26 15:34:27.729547844 (504h5m27.600630899s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.729589394 (504h5m27.600672439s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 15:34:27.729597139 (504h5m27.600680174s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.729978565 (504h5m27.60106162s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50458->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50458->45.91.116.11:4119: use of closed network connection Dec 26 15:34:27.730003902 (504h5m27.601086947s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.730035853 (504h5m27.601118898s) [seg_pos={0 0}, upload=442354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:34:27.730130644 (504h5m27.601213689s) [upload=442354]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:34:27.730136014 (504h5m27.601219049s) [upload=442354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:34:28.030174941 (504h5m27.901257976s) [upload=442354]: Done uploading segments... Dec 26 15:34:28.030196833 (504h5m27.901279868s) [upload=442354]: Upload finished. err: <nil> Duration: 30.003075461s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X.] duration: 3.405466032s trace: https://jaeger.dev.storj.io/trace/71972eeda6034f51 FAIL (2025-12-26 15:34:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.405493073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX..X.X..X..X.X...XX.XXXX.X..................X..........X.XX] duration: 1m0.002745529s trace: https://jaeger.dev.storj.io/trace/4d6ec53c41a9b5cc FAIL (2025-12-26 15:34:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, Piece ID: 54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA) context deadline exceeded; piecestore: (Node ID: 1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, Piece ID: 5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ) context deadline exceeded; piecestore: (Node ID: 1CMsUV4eQnyd4FLQYLi5xdPDurkVaL2kyn8EW4VigZsh8sdTv6, Piece ID: VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA) context deadline exceeded ======LOGS====== ODZ2M5DFJDIPA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.484026986 (504h4m44.355110031s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.484477945 (504h4m44.35556099s) [noise=true , address=77.90.31.19:4866 , piece_id=F32RITTYFC6EO5E7NTOTMNLJ2VMEHN46YOXPCWVG25VDPIP4B75Q, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485078588 (504h4m44.356161633s) [noise=true , address=45.91.116.4:4963 , piece_id=5LDE6TJMLXDAFPDJ7ZMKKTS6LS22VDZKW7MWMVEGA2DU37F2PDYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485615390 (504h4m44.356698435s) [noise=true , address=45.91.116.11:4121 , piece_id=5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485637873 (504h4m44.356720918s) [noise=true , address=145.79.227.44:4205 , piece_id=I5ISL5NWNUMQKLO75E7EJTHKN3R2NDZ5LDBH72DMO2PTQS67R2VA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486193952 (504h4m44.357276987s) [noise=true , address=77.90.31.14:4670 , piece_id=DRBNQDD6O3CHX5OYAMYMKS7SRI5A4QOFVOAEKI3GNF3JWUKC55JA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486453666 (504h4m44.357536701s) [noise=true , address=45.91.116.14:4201 , piece_id=IHFNLZRXMP3GFZP65PZSWG5GZ2ZFKTG2YVYC6GQ2BFEGQHS33HBA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486754038 (504h4m44.357837083s) [noise=true , address=64.49.13.170:4725 , piece_id=3XFNXHSV6HMV5IDZWXZ5XH7RWRYS2UXATNGZWQNTVY57DBB23GXQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487114945 (504h4m44.35819799s) [noise=true , address=145.79.227.29:4954 , piece_id=A6GARORLQWVZCU2GTFJ555RIXW7PCZB4I3CUJ6OXXCFIBJCJHBYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487331979 (504h4m44.358415024s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487769442 (504h4m44.358852487s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487874922 (504h4m44.358957967s) [noise=true , address=77.90.31.15:4703 , piece_id=54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.488454076 (504h4m44.359537111s) [noise=true , address=45.91.116.13:4174 , piece_id=VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.491854641 (504h4m44.362937686s) [noise=true , address=64.49.13.26:4945 , piece_id=7ISUES6U4NH3NZU3GUOR2IMXLVRTSPUSWUZNC2QICKZ3MJELN4HA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:45.797674129 (504h4m45.668757174s) [upload=442344]: Next returned nil segment Dec 26 15:33:45.797692585 (504h4m45.66877562s) [upload=442344]: Waiting for error group managing segments... Dec 26 15:33:46.130577946 (504h4m46.001660981s) [noise=true , address=193.149.164.143:4702 , piece_id=ORKNZ2J4IE7SSNJGTYPD5V4VKDJGGKXVE7Z7BW6ODZ2M5DFJDIPA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.205726252 (504h4m46.076809297s) [noise=true , address=64.49.13.26:4945 , piece_id=7ISUES6U4NH3NZU3GUOR2IMXLVRTSPUSWUZNC2QICKZ3MJELN4HA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.308550998 (504h4m46.179634043s) [noise=true , address=64.49.13.170:4725 , piece_id=3XFNXHSV6HMV5IDZWXZ5XH7RWRYS2UXATNGZWQNTVY57DBB23GXQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.577575107 (504h4m46.448658152s) [noise=true , address=45.91.116.14:4201 , piece_id=IHFNLZRXMP3GFZP65PZSWG5GZ2ZFKTG2YVYC6GQ2BFEGQHS33HBA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.615742895 (504h4m46.48682593s) [noise=true , address=45.91.116.11:4121 , piece_id=5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.616326656 (504h4m46.487409691s) [noise=true , address=45.91.116.13:4174 , piece_id=VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.699340129 (504h4m46.570423174s) [noise=true , address=45.91.116.4:4963 , piece_id=5LDE6TJMLXDAFPDJ7ZMKKTS6LS22VDZKW7MWMVEGA2DU37F2PDYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.708773672 (504h4m46.579856707s) [noise=true , address=145.79.227.29:4954 , piece_id=A6GARORLQWVZCU2GTFJ555RIXW7PCZB4I3CUJ6OXXCFIBJCJHBYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.955154153 (504h4m46.826237198s) [noise=true , address=77.90.31.15:4703 , piece_id=54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.142693852 (504h4m47.013776897s) [noise=true , address=77.90.31.19:4866 , piece_id=F32RITTYFC6EO5E7NTOTMNLJ2VMEHN46YOXPCWVG25VDPIP4B75Q, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.276150466 (504h4m47.147233511s) [noise=true , address=145.79.227.40:4053 , piece_id=2J2EY3X3WFXWWXZBHC6R2EN3SSOT35BJJULBJWLT577PJA6DY7CQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.304118842 (504h4m47.175201877s) [noise=true , address=145.79.227.44:4205 , piece_id=I5ISL5NWNUMQKLO75E7EJTHKN3R2NDZ5LDBH72DMO2PTQS67R2VA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.440569149 (504h4m47.311652194s) [noise=true , address=145.79.227.30:4796 , piece_id=MYHNLDAKFLPH2A7JEVQCAT7YQAO54XJ2SFD2MGQPPAFVIDVO47KA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.449458225 (504h4m47.32054128s) [noise=true , address=77.90.31.14:4670 , piece_id=DRBNQDD6O3CHX5OYAMYMKS7SRI5A4QOFVOAEKI3GNF3JWUKC55JA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.449479796 (504h4m47.320562841s) [seg_pos={0 0}, upload=442344]: Segment reached optimal threshold of 14 pieces. Dec 26 15:33:47.449793704 (504h4m47.320876739s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection; write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection Dec 26 15:33:47.449856813 (504h4m47.320939858s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.449994956 (504h4m47.321078001s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection Dec 26 15:33:47.450015105 (504h4m47.32109815s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450029161 (504h4m47.321112206s) [noise=true , address=77.90.31.11:4576 , piece_id=T5UX5RSTPFYOI6OS744JCEB4NNNBLYP24XCW7LOEYTDPTKHXAGAQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv): protocol: read tcp 188.166.225.170:37370->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:33:47.450036004 (504h4m47.321119039s) [noise=true , address=77.90.31.11:4576 , piece_id=T5UX5RSTPFYOI6OS744JCEB4NNNBLYP24XCW7LOEYTDPTKHXAGAQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450179719 (504h4m47.321262764s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 15:33:47.450190198 (504h4m47.321273243s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450218202 (504h4m47.321301247s) [seg_pos={0 0}, upload=442344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:33:47.450321839 (504h4m47.321404884s) [upload=442344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:33:47.450328582 (504h4m47.321411637s) [upload=442344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:33:47.743361678 (504h4m47.614444733s) [upload=442344]: Done uploading segments... Dec 26 15:33:47.743387397 (504h4m47.614470442s) [upload=442344]: Upload finished. err: <nil> Duration: 1m0.00276163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....XX..X.X.....X.....X.X............XXX....XX...X......X..] duration: 7.011270652s trace: https://jaeger.dev.storj.io/trace/db5d56fbf5ab72f FAIL (2025-12-26 15:34:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 327]: Starting upload Dec 26 15:32:27.150869037 (504h3m27.021952082s) [upload=442327]: Done starting upload Dec 26 15:32:27.150966753 (504h3m27.022049798s) [upload=442327]: Uploading segments... Dec 26 15:32:27.151111518 (504h3m27.022194563s) [upload=442327]: Got next segment. Inline: false Dec 26 15:32:27.151160903 (504h3m27.022243938s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:32:27.151170230 (504h3m27.022253265s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:32:27.396820902 (504h3m27.267903947s) [seg_pos={0 0}, upload=442327]: Begin upload segment... Dec 26 15:32:27.396878382 (504h3m27.267961427s) [seg_pos={0 0}, upload=442327]: Done begin upload segment. Dec 26 15:32:27.396911815 (504h3m27.267994851s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397064496 (504h3m27.268147531s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397617349 (504h3m27.268700414s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398128021 (504h3m27.269211056s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398146155 (504h3m27.26922919s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398458009 (504h3m27.269541044s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398467878 (504h3m27.269550923s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398732943 (504h3m27.269815978s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398758772 (504h3m27.269841807s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399028857 (504h3m27.270111902s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399043615 (504h3m27.27012666s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399316584 (504h3m27.27039963s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399323958 (504h3m27.270407003s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399593432 (504h3m27.270676467s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399869507 (504h3m27.270952543s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.400755334 (504h3m27.271838369s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:29.102300563 (504h3m28.973383608s) [upload=442327]: Next returned nil segment Dec 26 15:32:29.102325200 (504h3m28.973408235s) [upload=442327]: Waiting for error group managing segments... Dec 26 15:32:29.388594837 (504h3m29.259677872s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.389572970 (504h3m29.260656015s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.400604706 (504h3m29.271687741s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.403736069 (504h3m29.274819114s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.404653175 (504h3m29.27573621s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.407490368 (504h3m29.278573403s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.408188768 (504h3m29.279271803s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410656246 (504h3m29.281739281s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410946158 (504h3m29.282029193s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.413745288 (504h3m29.284828334s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.425009076 (504h3m29.296092111s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.670341572 (504h3m29.541424607s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713495595 (504h3m29.58457863s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713513468 (504h3m29.584596514s) [seg_pos={0 0}, upload=442327]: Segment reached optimal threshold of 13 pieces. Dec 26 15:32:29.713767834 (504h3m29.584850879s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection Dec 26 15:32:29.713777161 (504h3m29.584860206s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714024021 (504h3m29.585107066s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:38774->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:32:29.714034621 (504h3m29.585117666s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714088314 (504h3m29.585171359s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:29.714097962 (504h3m29.585181007s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714116087 (504h3m29.585199122s) [seg_pos={0 0}, upload=442327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:32:29.714198333 (504h3m29.585281408s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:32:29.714207089 (504h3m29.585290124s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:32:30.011069438 (504h3m29.882152473s) [upload=442327]: Done uploading segments... Dec 26 15:32:30.011090958 (504h3m29.882174003s) [upload=442327]: Upload finished. err: <nil> Duration: 7.011403918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX..X.X..X..X.X...XX.XXXX.X..................X..........X.X] duration: 1m0.001729695s trace: https://jaeger.dev.storj.io/trace/61dc4d67ec619425 FAIL (2025-12-26 15:33:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, Piece ID: XKARWE73SY3O2NEUUERD5IJ2RJ3LNNNTQERYZOERRB42RNVOB4EA) context deadline exceeded; piecestore: (Node ID: 1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, Piece ID: DAJ7JX7MMPCOOVWLZMKSTJJCGZEVEQWXRUSKKXDLE6Q4WYDGDVYA) context deadline exceeded; piecestore: (Node ID: 122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, Piece ID: NBLKWG5PLHW2SQ563RBNNLIEH6ZBNLZRHPYPPFH26XK6W5DHXVQQ) context deadline exceeded ======LOGS====== 6 15:32:44.260628716 (504h3m44.131711761s) [noise=true , address=145.79.227.47:4281 , piece_id=LE7IK27V5WYDLT6B6PZOLOLO6U37SZPXPLDDD6TO7LFSS7IPC65Q, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.260940239 (504h3m44.132023284s) [noise=true , address=45.91.116.4:4960 , piece_id=TCCJBCOL23W7BNAPUQ3K6G5VR6EP2KRHDVWRZJOOAK2GX36IIWMA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.261208099 (504h3m44.132291134s) [noise=true , address=145.79.227.34:4927 , piece_id=MHQ5HOSRWVH4MN5PQC6ABC2XZWZ3BMAJ6BOW4PVEZ352X3IJR3YQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.261713852 (504h3m44.132796897s) [noise=true , address=145.79.227.44:4227 , piece_id=PMANVYTX3G3AQHIJSXSAYRI4FSF56JUT5BQK6QZZKPFT2Z6PX4TA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.262188957 (504h3m44.133272002s) [noise=true , address=45.91.116.10:4097 , piece_id=HOWSTA5KFZ7NGRXZPDDSZD2BFFFDZTJUTJL3GEWZ5GHCKB2AUTFQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.263588081 (504h3m44.134671126s) [noise=true , address=77.90.31.19:4873 , piece_id=GXZZHUADT4462UKIFXSEZMVIUJ32J4CTQIQAJYI5ATJJM6AEF5WA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.264038048 (504h3m44.135121093s) [noise=true , address=145.79.227.30:4785 , piece_id=EVFF6J2FLJNH6YGLWDGJCPCSJTLWM473ZJVLJDTT4OF4XYGGQPUQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.264488996 (504h3m44.135572041s) [noise=true , address=77.90.31.11:4574 , piece_id=DAJ7JX7MMPCOOVWLZMKSTJJCGZEVEQWXRUSKKXDLE6Q4WYDGDVYA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.266184726 (504h3m44.137267771s) [noise=true , address=77.90.31.21:4908 , piece_id=4WCHAGG4O2VVVXZZSJC2H5XQADPVZHQJHUQYPMAI4GI2Z3CWE7HQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.267910661 (504h3m44.138993707s) [noise=true , address=145.79.227.40:4059 , piece_id=WOW4SDDHF2FV3472XE6LLFWJOL2LCVGWTYOJ6ZECDY5OQK6LXGTA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.269608244 (504h3m44.140691289s) [noise=true , address=77.90.31.10:4529 , piece_id=NBLKWG5PLHW2SQ563RBNNLIEH6ZBNLZRHPYPPFH26XK6W5DHXVQQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.270943066 (504h3m44.142026101s) [noise=true , address=77.90.31.18:4820 , piece_id=XKARWE73SY3O2NEUUERD5IJ2RJ3LNNNTQERYZOERRB42RNVOB4EA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.272213135 (504h3m44.14329617s) [noise=true , address=64.49.13.169:4706 , piece_id=PY4667A2DCONGTD2CGFZ6I3DZON3JGBPEDAFRECAWGPND7S2P45A, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.273524422 (504h3m44.144607467s) [noise=true , address=77.90.31.17:4792 , piece_id=SEZ64NSDSE7PFCGOWHVZSPJHYOXK4AGUJTTO6YBYXMYPSZH2PBOQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.274861838 (504h3m44.145944873s) [noise=true , address=77.90.31.16:4768 , piece_id=KAIFWFKT36FG2672SRWVHB3KF3S2X2JTXJCIAMW5VBMHZUVUDFRQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.275434809 (504h3m44.146517854s) [noise=true , address=145.79.227.43:4175 , piece_id=TDHCMWDBTBTJQGDW3UDSIAEZPAHDCGUNOALAAJUXM2DY7KYKIJIQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:45.735001559 (504h3m45.606084594s) [upload=442332]: Next returned nil segment Dec 26 15:32:45.735015345 (504h3m45.60609839s) [upload=442332]: Waiting for error group managing segments... Dec 26 15:32:46.157631319 (504h3m46.028714364s) [noise=true , address=145.79.227.47:4281 , piece_id=LE7IK27V5WYDLT6B6PZOLOLO6U37SZPXPLDDD6TO7LFSS7IPC65Q, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.164758311 (504h3m46.035841346s) [noise=true , address=145.79.227.43:4175 , piece_id=TDHCMWDBTBTJQGDW3UDSIAEZPAHDCGUNOALAAJUXM2DY7KYKIJIQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.320405691 (504h3m46.191488726s) [noise=true , address=77.90.31.21:4908 , piece_id=4WCHAGG4O2VVVXZZSJC2H5XQADPVZHQJHUQYPMAI4GI2Z3CWE7HQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.359214530 (504h3m46.230297575s) [noise=true , address=77.90.31.11:4574 , piece_id=DAJ7JX7MMPCOOVWLZMKSTJJCGZEVEQWXRUSKKXDLE6Q4WYDGDVYA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.396758129 (504h3m46.267841164s) [noise=true , address=45.91.116.4:4960 , piece_id=TCCJBCOL23W7BNAPUQ3K6G5VR6EP2KRHDVWRZJOOAK2GX36IIWMA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.438396866 (504h3m46.309479911s) [noise=true , address=77.90.31.16:4768 , piece_id=KAIFWFKT36FG2672SRWVHB3KF3S2X2JTXJCIAMW5VBMHZUVUDFRQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.483980976 (504h3m46.355064021s) [noise=true , address=77.90.31.19:4873 , piece_id=GXZZHUADT4462UKIFXSEZMVIUJ32J4CTQIQAJYI5ATJJM6AEF5WA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.494760142 (504h3m46.365843187s) [noise=true , address=77.90.31.10:4529 , piece_id=NBLKWG5PLHW2SQ563RBNNLIEH6ZBNLZRHPYPPFH26XK6W5DHXVQQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.741890932 (504h3m46.612973967s) [noise=true , address=77.90.31.18:4820 , piece_id=XKARWE73SY3O2NEUUERD5IJ2RJ3LNNNTQERYZOERRB42RNVOB4EA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.000445775 (504h3m46.87152883s) [noise=true , address=77.90.31.13:4625 , piece_id=I7KZWLCK7GLZYN6JIBIH4BY3OW4HXR7HH6YLMUQWP5LELNXEAQAA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.092510673 (504h3m46.963593708s) [noise=true , address=145.79.227.44:4227 , piece_id=PMANVYTX3G3AQHIJSXSAYRI4FSF56JUT5BQK6QZZKPFT2Z6PX4TA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.096262397 (504h3m46.967345452s) [noise=true , address=145.79.227.30:4785 , piece_id=EVFF6J2FLJNH6YGLWDGJCPCSJTLWM473ZJVLJDTT4OF4XYGGQPUQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.104195012 (504h3m46.975278047s) [noise=true , address=145.79.227.40:4059 , piece_id=WOW4SDDHF2FV3472XE6LLFWJOL2LCVGWTYOJ6ZECDY5OQK6LXGTA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.130868082 (504h3m47.001951117s) [noise=true , address=145.79.227.34:4927 , piece_id=MHQ5HOSRWVH4MN5PQC6ABC2XZWZ3BMAJ6BOW4PVEZ352X3IJR3YQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.130886237 (504h3m47.001969272s) [seg_pos={0 0}, upload=442332]: Segment reached optimal threshold of 14 pieces. Dec 26 15:32:47.131077551 (504h3m47.002160586s) [noise=true , address=77.90.31.17:4792 , piece_id=SEZ64NSDSE7PFCGOWHVZSPJHYOXK4AGUJTTO6YBYXMYPSZH2PBOQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:47.131083923 (504h3m47.002166958s) [noise=true , address=77.90.31.17:4792 , piece_id=SEZ64NSDSE7PFCGOWHVZSPJHYOXK4AGUJTTO6YBYXMYPSZH2PBOQ, seg_pos={0 0}, upload=442332]: Long tail cancellation due to optimal threshold Dec 26 15:32:47.131194734 (504h3m47.002277779s) [noise=true , address=45.91.116.10:4097 , piece_id=HOWSTA5KFZ7NGRXZPDDSZD2BFFFDZTJUTJL3GEWZ5GHCKB2AUTFQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:47.131200846 (504h3m47.002283881s) [noise=true , address=45.91.116.10:4097 , piece_id=HOWSTA5KFZ7NGRXZPDDSZD2BFFFDZTJUTJL3GEWZ5GHCKB2AUTFQ, seg_pos={0 0}, upload=442332]: Long tail cancellation due to optimal threshold Dec 26 15:32:47.131313601 (504h3m47.002396646s) [noise=true , address=64.49.13.139:4473 , piece_id=KGXLN6OSUDXEIQ6CQG3UFQS3OKUBOLZX2UYANBWJODB2UILUEHIA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vE9uZrGbtGAPxJtzwoowofHwtp3d5MeeJw7qzAtSCdzcrxzKf): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:47.131335142 (504h3m47.002418187s) [noise=true , address=64.49.13.139:4473 , piece_id=KGXLN6OSUDXEIQ6CQG3UFQS3OKUBOLZX2UYANBWJODB2UILUEHIA, seg_pos={0 0}, upload=442332]: Long tail cancellation due to optimal threshold Dec 26 15:32:47.131370139 (504h3m47.002453184s) [noise=true , address=64.49.13.169:4706 , piece_id=PY4667A2DCONGTD2CGFZ6I3DZON3JGBPEDAFRECAWGPND7S2P45A, seg_pos={0 0}, upload=442332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:42364->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:42364->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42364->64.49.13.169:4706: use of closed network connection Dec 26 15:32:47.131392120 (504h3m47.002475165s) [noise=true , address=64.49.13.169:4706 , piece_id=PY4667A2DCONGTD2CGFZ6I3DZON3JGBPEDAFRECAWGPND7S2P45A, seg_pos={0 0}, upload=442332]: Long tail cancellation due to optimal threshold Dec 26 15:32:47.131426355 (504h3m47.00250939s) [seg_pos={0 0}, upload=442332]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:32:47.131518822 (504h3m47.002601867s) [upload=442332]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:32:47.131526266 (504h3m47.002609301s) [upload=442332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:32:47.444749867 (504h3m47.315832902s) [upload=442332]: Done uploading segments... Dec 26 15:32:47.444776098 (504h3m47.315859143s) [upload=442332]: Upload finished. err: <nil> Duration: 1m0.001750624s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X] duration: 5.001040845s trace: https://jaeger.dev.storj.io/trace/71972eeda6034f51 FAIL (2025-12-26 15:33:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001063358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......XX..X.X.....X.....X.X............XXX....XX...X......X.] duration: 5.841773095s trace: https://jaeger.dev.storj.io/trace/db5d56fbf5ab72f FAIL (2025-12-26 15:33:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 327]: Starting upload Dec 26 15:32:27.150869037 (504h3m27.021952082s) [upload=442327]: Done starting upload Dec 26 15:32:27.150966753 (504h3m27.022049798s) [upload=442327]: Uploading segments... Dec 26 15:32:27.151111518 (504h3m27.022194563s) [upload=442327]: Got next segment. Inline: false Dec 26 15:32:27.151160903 (504h3m27.022243938s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:32:27.151170230 (504h3m27.022253265s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:32:27.396820902 (504h3m27.267903947s) [seg_pos={0 0}, upload=442327]: Begin upload segment... Dec 26 15:32:27.396878382 (504h3m27.267961427s) [seg_pos={0 0}, upload=442327]: Done begin upload segment. Dec 26 15:32:27.396911815 (504h3m27.267994851s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397064496 (504h3m27.268147531s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397617349 (504h3m27.268700414s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398128021 (504h3m27.269211056s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398146155 (504h3m27.26922919s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398458009 (504h3m27.269541044s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398467878 (504h3m27.269550923s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398732943 (504h3m27.269815978s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398758772 (504h3m27.269841807s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399028857 (504h3m27.270111902s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399043615 (504h3m27.27012666s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399316584 (504h3m27.27039963s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399323958 (504h3m27.270407003s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399593432 (504h3m27.270676467s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399869507 (504h3m27.270952543s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.400755334 (504h3m27.271838369s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:29.102300563 (504h3m28.973383608s) [upload=442327]: Next returned nil segment Dec 26 15:32:29.102325200 (504h3m28.973408235s) [upload=442327]: Waiting for error group managing segments... Dec 26 15:32:29.388594837 (504h3m29.259677872s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.389572970 (504h3m29.260656015s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.400604706 (504h3m29.271687741s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.403736069 (504h3m29.274819114s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.404653175 (504h3m29.27573621s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.407490368 (504h3m29.278573403s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.408188768 (504h3m29.279271803s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410656246 (504h3m29.281739281s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410946158 (504h3m29.282029193s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.413745288 (504h3m29.284828334s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.425009076 (504h3m29.296092111s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.670341572 (504h3m29.541424607s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713495595 (504h3m29.58457863s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713513468 (504h3m29.584596514s) [seg_pos={0 0}, upload=442327]: Segment reached optimal threshold of 13 pieces. Dec 26 15:32:29.713767834 (504h3m29.584850879s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection Dec 26 15:32:29.713777161 (504h3m29.584860206s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714024021 (504h3m29.585107066s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:38774->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:32:29.714034621 (504h3m29.585117666s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714088314 (504h3m29.585171359s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:29.714097962 (504h3m29.585181007s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714116087 (504h3m29.585199122s) [seg_pos={0 0}, upload=442327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:32:29.714198333 (504h3m29.585281408s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:32:29.714207089 (504h3m29.585290124s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:32:30.011069438 (504h3m29.882152473s) [upload=442327]: Done uploading segments... Dec 26 15:32:30.011090958 (504h3m29.882174003s) [upload=442327]: Upload finished. err: <nil> Duration: 5.841800847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......XX..X.X.....X.....X.X............XXX....XX...X......X] duration: 30.003293848s trace: https://jaeger.dev.storj.io/trace/db5d56fbf5ab72f FAIL (2025-12-26 15:32:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 327]: Starting upload Dec 26 15:32:27.150869037 (504h3m27.021952082s) [upload=442327]: Done starting upload Dec 26 15:32:27.150966753 (504h3m27.022049798s) [upload=442327]: Uploading segments... Dec 26 15:32:27.151111518 (504h3m27.022194563s) [upload=442327]: Got next segment. Inline: false Dec 26 15:32:27.151160903 (504h3m27.022243938s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:32:27.151170230 (504h3m27.022253265s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:32:27.396820902 (504h3m27.267903947s) [seg_pos={0 0}, upload=442327]: Begin upload segment... Dec 26 15:32:27.396878382 (504h3m27.267961427s) [seg_pos={0 0}, upload=442327]: Done begin upload segment. Dec 26 15:32:27.396911815 (504h3m27.267994851s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397064496 (504h3m27.268147531s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397617349 (504h3m27.268700414s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398128021 (504h3m27.269211056s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398146155 (504h3m27.26922919s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398458009 (504h3m27.269541044s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398467878 (504h3m27.269550923s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398732943 (504h3m27.269815978s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398758772 (504h3m27.269841807s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399028857 (504h3m27.270111902s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399043615 (504h3m27.27012666s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399316584 (504h3m27.27039963s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399323958 (504h3m27.270407003s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399593432 (504h3m27.270676467s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399869507 (504h3m27.270952543s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.400755334 (504h3m27.271838369s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:29.102300563 (504h3m28.973383608s) [upload=442327]: Next returned nil segment Dec 26 15:32:29.102325200 (504h3m28.973408235s) [upload=442327]: Waiting for error group managing segments... Dec 26 15:32:29.388594837 (504h3m29.259677872s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.389572970 (504h3m29.260656015s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.400604706 (504h3m29.271687741s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.403736069 (504h3m29.274819114s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.404653175 (504h3m29.27573621s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.407490368 (504h3m29.278573403s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.408188768 (504h3m29.279271803s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410656246 (504h3m29.281739281s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410946158 (504h3m29.282029193s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.413745288 (504h3m29.284828334s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.425009076 (504h3m29.296092111s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.670341572 (504h3m29.541424607s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713495595 (504h3m29.58457863s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713513468 (504h3m29.584596514s) [seg_pos={0 0}, upload=442327]: Segment reached optimal threshold of 13 pieces. Dec 26 15:32:29.713767834 (504h3m29.584850879s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection Dec 26 15:32:29.713777161 (504h3m29.584860206s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714024021 (504h3m29.585107066s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:38774->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:32:29.714034621 (504h3m29.585117666s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714088314 (504h3m29.585171359s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:29.714097962 (504h3m29.585181007s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714116087 (504h3m29.585199122s) [seg_pos={0 0}, upload=442327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:32:29.714198333 (504h3m29.585281408s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:32:29.714207089 (504h3m29.585290124s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:32:30.011069438 (504h3m29.882152473s) [upload=442327]: Done uploading segments... Dec 26 15:32:30.011090958 (504h3m29.882174003s) [upload=442327]: Upload finished. err: <nil> Duration: 30.00331642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.] duration: 5.001126608s trace: https://jaeger.dev.storj.io/trace/421007474598b989 FAIL (2025-12-26 15:32:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001156124s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)