Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....XXX....X....X.XX..X....XX.XX..................X...X..] duration: 6.460105304s trace: https://jaeger.dev.storj.io/trace/2d666678f6513905 FAIL (2025-12-30 08:00:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q) context deadline exceeded ======LOGS====== Dec 30 07:59:10.202653803 (592h30m10.073736848s) [upload=519423]: Starting upload Dec 30 07:59:10.202752560 (592h30m10.073835605s) [upload=519423]: Done starting upload Dec 30 07:59:10.202990363 (592h30m10.074073418s) [upload=519423]: Uploading segments... Dec 30 07:59:10.203134788 (592h30m10.074217833s) [upload=519423]: Got next segment. Inline: false Dec 30 07:59:10.203202307 (592h30m10.074285342s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:59:10.203208508 (592h30m10.074291553s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:59:10.447546036 (592h30m10.318629081s) [seg_pos={0 0}, upload=519423]: Begin upload segment... Dec 30 07:59:10.447642639 (592h30m10.318725684s) [seg_pos={0 0}, upload=519423]: Done begin upload segment. Dec 30 07:59:10.447706581 (592h30m10.318789626s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447713214 (592h30m10.318796249s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447924818 (592h30m10.319007863s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448485695 (592h30m10.31956873s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448958796 (592h30m10.320041841s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.449438610 (592h30m10.320521655s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450059773 (592h30m10.321142818s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450651269 (592h30m10.321734314s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.452118853 (592h30m10.323201898s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453400123 (592h30m10.324483168s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453661260 (592h30m10.324744305s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.455123524 (592h30m10.326206559s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.456465690 (592h30m10.327548726s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.458194782 (592h30m10.329277817s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.460128836 (592h30m10.331211881s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.462118644 (592h30m10.333201679s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:12.144433349 (592h30m12.015516395s) [upload=519423]: Next returned nil segment Dec 30 07:59:12.144446866 (592h30m12.015529911s) [upload=519423]: Waiting for error group managing segments... Dec 30 07:59:12.427594743 (592h30m12.298677788s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.433067574 (592h30m12.304150619s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.450044672 (592h30m12.321127717s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.456286157 (592h30m12.327369192s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.460033563 (592h30m12.331116598s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.463362521 (592h30m12.334445556s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.467905432 (592h30m12.338988467s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.468015401 (592h30m12.339098446s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.534010025 (592h30m12.405093061s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.553978358 (592h30m12.425061393s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.653048103 (592h30m12.524131158s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.876260555 (592h30m12.7473436s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265833353 (592h30m13.136916388s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265855445 (592h30m13.13693848s) [seg_pos={0 0}, upload=519423]: Segment reached optimal threshold of 13 pieces. Dec 30 07:59:13.266178440 (592h30m13.137261475s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:46954->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:59:13.266190233 (592h30m13.137273308s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266311203 (592h30m13.137394238s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266327554 (592h30m13.137410589s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266376197 (592h30m13.137459232s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266382398 (592h30m13.137465443s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266397808 (592h30m13.137480843s) [seg_pos={0 0}, upload=519423]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:59:13.266510162 (592h30m13.137593207s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:59:13.266518638 (592h30m13.137601673s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:59:13.555405394 (592h30m13.426488439s) [upload=519423]: Done uploading segments... Dec 30 07:59:13.555425743 (592h30m13.426508778s) [upload=519423]: Upload finished. err: <nil> Duration: 6.460335992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 42.607760659s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 42.607780587s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X.] duration: 4.367117796s trace: https://jaeger.dev.storj.io/trace/781ac7d52d9c13dc FAIL (2025-12-30 08:00:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.367148523s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X] duration: 5.00223422s trace: https://jaeger.dev.storj.io/trace/781ac7d52d9c13dc FAIL (2025-12-30 08:00:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002259108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X....XXX....X....X.XX..X....XX.XX..................X...X.] duration: 5.912237752s trace: https://jaeger.dev.storj.io/trace/2d666678f6513905 FAIL (2025-12-30 07:59:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q) context deadline exceeded ======LOGS====== Dec 30 07:59:10.202653803 (592h30m10.073736848s) [upload=519423]: Starting upload Dec 30 07:59:10.202752560 (592h30m10.073835605s) [upload=519423]: Done starting upload Dec 30 07:59:10.202990363 (592h30m10.074073418s) [upload=519423]: Uploading segments... Dec 30 07:59:10.203134788 (592h30m10.074217833s) [upload=519423]: Got next segment. Inline: false Dec 30 07:59:10.203202307 (592h30m10.074285342s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:59:10.203208508 (592h30m10.074291553s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:59:10.447546036 (592h30m10.318629081s) [seg_pos={0 0}, upload=519423]: Begin upload segment... Dec 30 07:59:10.447642639 (592h30m10.318725684s) [seg_pos={0 0}, upload=519423]: Done begin upload segment. Dec 30 07:59:10.447706581 (592h30m10.318789626s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447713214 (592h30m10.318796249s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447924818 (592h30m10.319007863s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448485695 (592h30m10.31956873s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448958796 (592h30m10.320041841s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.449438610 (592h30m10.320521655s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450059773 (592h30m10.321142818s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450651269 (592h30m10.321734314s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.452118853 (592h30m10.323201898s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453400123 (592h30m10.324483168s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453661260 (592h30m10.324744305s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.455123524 (592h30m10.326206559s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.456465690 (592h30m10.327548726s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.458194782 (592h30m10.329277817s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.460128836 (592h30m10.331211881s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.462118644 (592h30m10.333201679s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:12.144433349 (592h30m12.015516395s) [upload=519423]: Next returned nil segment Dec 30 07:59:12.144446866 (592h30m12.015529911s) [upload=519423]: Waiting for error group managing segments... Dec 30 07:59:12.427594743 (592h30m12.298677788s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.433067574 (592h30m12.304150619s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.450044672 (592h30m12.321127717s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.456286157 (592h30m12.327369192s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.460033563 (592h30m12.331116598s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.463362521 (592h30m12.334445556s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.467905432 (592h30m12.338988467s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.468015401 (592h30m12.339098446s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.534010025 (592h30m12.405093061s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.553978358 (592h30m12.425061393s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.653048103 (592h30m12.524131158s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.876260555 (592h30m12.7473436s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265833353 (592h30m13.136916388s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265855445 (592h30m13.13693848s) [seg_pos={0 0}, upload=519423]: Segment reached optimal threshold of 13 pieces. Dec 30 07:59:13.266178440 (592h30m13.137261475s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:46954->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:59:13.266190233 (592h30m13.137273308s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266311203 (592h30m13.137394238s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266327554 (592h30m13.137410589s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266376197 (592h30m13.137459232s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266382398 (592h30m13.137465443s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266397808 (592h30m13.137480843s) [seg_pos={0 0}, upload=519423]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:59:13.266510162 (592h30m13.137593207s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:59:13.266518638 (592h30m13.137601673s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:59:13.555405394 (592h30m13.426488439s) [upload=519423]: Done uploading segments... Dec 30 07:59:13.555425743 (592h30m13.426508778s) [upload=519423]: Upload finished. err: <nil> Duration: 5.912261248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 30.291986372s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:59:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 30.29200206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X...X....XXX....X....X.XX..X....XX.XX..................X...X] duration: 30.004926816s trace: https://jaeger.dev.storj.io/trace/2d666678f6513905 FAIL (2025-12-30 07:59:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q) context deadline exceeded ======LOGS====== Dec 30 07:59:10.202653803 (592h30m10.073736848s) [upload=519423]: Starting upload Dec 30 07:59:10.202752560 (592h30m10.073835605s) [upload=519423]: Done starting upload Dec 30 07:59:10.202990363 (592h30m10.074073418s) [upload=519423]: Uploading segments... Dec 30 07:59:10.203134788 (592h30m10.074217833s) [upload=519423]: Got next segment. Inline: false Dec 30 07:59:10.203202307 (592h30m10.074285342s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:59:10.203208508 (592h30m10.074291553s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:59:10.447546036 (592h30m10.318629081s) [seg_pos={0 0}, upload=519423]: Begin upload segment... Dec 30 07:59:10.447642639 (592h30m10.318725684s) [seg_pos={0 0}, upload=519423]: Done begin upload segment. Dec 30 07:59:10.447706581 (592h30m10.318789626s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447713214 (592h30m10.318796249s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447924818 (592h30m10.319007863s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448485695 (592h30m10.31956873s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448958796 (592h30m10.320041841s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.449438610 (592h30m10.320521655s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450059773 (592h30m10.321142818s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450651269 (592h30m10.321734314s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.452118853 (592h30m10.323201898s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453400123 (592h30m10.324483168s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453661260 (592h30m10.324744305s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.455123524 (592h30m10.326206559s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.456465690 (592h30m10.327548726s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.458194782 (592h30m10.329277817s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.460128836 (592h30m10.331211881s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.462118644 (592h30m10.333201679s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:12.144433349 (592h30m12.015516395s) [upload=519423]: Next returned nil segment Dec 30 07:59:12.144446866 (592h30m12.015529911s) [upload=519423]: Waiting for error group managing segments... Dec 30 07:59:12.427594743 (592h30m12.298677788s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.433067574 (592h30m12.304150619s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.450044672 (592h30m12.321127717s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.456286157 (592h30m12.327369192s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.460033563 (592h30m12.331116598s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.463362521 (592h30m12.334445556s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.467905432 (592h30m12.338988467s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.468015401 (592h30m12.339098446s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.534010025 (592h30m12.405093061s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.553978358 (592h30m12.425061393s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.653048103 (592h30m12.524131158s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.876260555 (592h30m12.7473436s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265833353 (592h30m13.136916388s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265855445 (592h30m13.13693848s) [seg_pos={0 0}, upload=519423]: Segment reached optimal threshold of 13 pieces. Dec 30 07:59:13.266178440 (592h30m13.137261475s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:46954->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:59:13.266190233 (592h30m13.137273308s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266311203 (592h30m13.137394238s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266327554 (592h30m13.137410589s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266376197 (592h30m13.137459232s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266382398 (592h30m13.137465443s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266397808 (592h30m13.137480843s) [seg_pos={0 0}, upload=519423]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:59:13.266510162 (592h30m13.137593207s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:59:13.266518638 (592h30m13.137601673s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:59:13.555405394 (592h30m13.426488439s) [upload=519423]: Done uploading segments... Dec 30 07:59:13.555425743 (592h30m13.426508778s) [upload=519423]: Upload finished. err: <nil> Duration: 30.004951031s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..] duration: 3.542754909s trace: https://jaeger.dev.storj.io/trace/2754cbfb05a23cc9 FAIL (2025-12-30 07:59:29 UTC) orbiter warning: context deadline exceeded Duration: 3.542903612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 37.411672174s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:59:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 37.411692783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X.] duration: 3.391076328s trace: https://jaeger.dev.storj.io/trace/2754cbfb05a23cc9 FAIL (2025-12-30 07:59:01 UTC) orbiter warning: context deadline exceeded Duration: 3.391138807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 33.847320169s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:58:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 33.847350888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X...X....XXX....X....X.XX..X....XX.XX..................X...] duration: 5.828108394s trace: https://jaeger.dev.storj.io/trace/11cba5b7624f3db7 FAIL (2025-12-30 07:58:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ) context deadline exceeded ======LOGS====== Dec 30 07:56:08.009867651 (592h27m7.880950686s) [upload=519379]: Starting upload Dec 30 07:56:08.009985315 (592h27m7.88106836s) [upload=519379]: Done starting upload Dec 30 07:56:08.010211505 (592h27m7.88129455s) [upload=519379]: Uploading segments... Dec 30 07:56:08.010277080 (592h27m7.881360125s) [upload=519379]: Got next segment. Inline: false Dec 30 07:56:08.010318679 (592h27m7.881401724s) [upload=519379]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:56:08.010323990 (592h27m7.881407035s) [upload=519379]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:56:08.255992922 (592h27m8.127075957s) [seg_pos={0 0}, upload=519379]: Begin upload segment... Dec 30 07:56:08.256053938 (592h27m8.127136973s) [seg_pos={0 0}, upload=519379]: Done begin upload segment. Dec 30 07:56:08.256102701 (592h27m8.127185746s) [noise=true , address=145.79.227.9:4365 , piece_id=SGCMBPUCE5MGMLCKAJOIKZY6VFNRDWI2OTUEHRW4HAI5Q5PGH6JA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256139631 (592h27m8.127222666s) [noise=true , address=145.79.227.43:4157 , piece_id=R5IIHK7IJ57X6R3XRYL4ZF54AKHYAGMF5AMNXDDXHOGXCEX7LLSA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256244531 (592h27m8.127327576s) [noise=true , address=145.79.227.3:4151 , piece_id=MX3WNFBFKBTHIUFURZMDC3JYM4VOZ4QALOYCOVEZAHSMEGFXY4DQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256794608 (592h27m8.127877643s) [noise=true , address=145.79.227.21:4475 , piece_id=KEIPC2WXXWCBYYFYEU6NOP7LV4PMQEN6PFEE57JIK2VRPB2XDVOQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257328394 (592h27m8.128411429s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257403759 (592h27m8.128486804s) [noise=true , address=145.79.227.2:4113 , piece_id=QSERPUV3HAF4VSVMANQGCHBA47JK5PXWNFGSCCFWN55RV5U5DUSQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257613338 (592h27m8.128696383s) [noise=true , address=145.79.227.50:4373 , piece_id=3CX6EMA7L755JW4LFSB3WGTKHM64QXYQ6WSSTMC45JTL5KCDOFBA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257910924 (592h27m8.128993959s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258200745 (592h27m8.129283781s) [noise=true , address=45.91.116.9:4262 , piece_id=FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258510836 (592h27m8.129593881s) [noise=true , address=145.79.227.11:4730 , piece_id=2HYNA6JVHGRQVIS5GOGA2FEQCUJPJVULHXOOLEU6PDR6EIN3O4TQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258883516 (592h27m8.129966551s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259179260 (592h27m8.130262305s) [noise=true , address=145.79.227.52:4445 , piece_id=URXZDZBV3XWVKFYTCC4TRZRBH6M6LZASCF7XUINIIWGPLPEJL2LQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259474712 (592h27m8.130557747s) [noise=true , address=45.91.116.8:4047 , piece_id=GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259848965 (592h27m8.13093201s) [noise=true , address=77.90.31.16:4735 , piece_id=BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.260132514 (592h27m8.131215559s) [noise=true , address=145.79.227.23:4546 , piece_id=27ZFFTLCU2FWVLBTKZBSG7UAXZTFSPJUPG6E32E5YH7PRZGTGGIA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.260712078 (592h27m8.131795113s) [noise=true , address=77.90.31.13:4629 , piece_id=N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:09.952650005 (592h27m9.82373305s) [upload=519379]: Next returned nil segment Dec 30 07:56:09.952662369 (592h27m9.823745404s) [upload=519379]: Waiting for error group managing segments... Dec 30 07:56:10.237249577 (592h27m10.108332643s) [noise=true , address=145.79.227.50:4373 , piece_id=3CX6EMA7L755JW4LFSB3WGTKHM64QXYQ6WSSTMC45JTL5KCDOFBA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.237828901 (592h27m10.108911946s) [noise=true , address=145.79.227.3:4151 , piece_id=MX3WNFBFKBTHIUFURZMDC3JYM4VOZ4QALOYCOVEZAHSMEGFXY4DQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.246031661 (592h27m10.117114706s) [noise=true , address=145.79.227.2:4113 , piece_id=QSERPUV3HAF4VSVMANQGCHBA47JK5PXWNFGSCCFWN55RV5U5DUSQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.250097562 (592h27m10.121180607s) [noise=true , address=145.79.227.21:4475 , piece_id=KEIPC2WXXWCBYYFYEU6NOP7LV4PMQEN6PFEE57JIK2VRPB2XDVOQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.250721180 (592h27m10.121804225s) [noise=true , address=145.79.227.52:4445 , piece_id=URXZDZBV3XWVKFYTCC4TRZRBH6M6LZASCF7XUINIIWGPLPEJL2LQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.276428340 (592h27m10.147511385s) [noise=true , address=145.79.227.11:4730 , piece_id=2HYNA6JVHGRQVIS5GOGA2FEQCUJPJVULHXOOLEU6PDR6EIN3O4TQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.285513861 (592h27m10.156596906s) [noise=true , address=145.79.227.9:4365 , piece_id=SGCMBPUCE5MGMLCKAJOIKZY6VFNRDWI2OTUEHRW4HAI5Q5PGH6JA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.333682441 (592h27m10.204765486s) [noise=true , address=145.79.227.23:4546 , piece_id=27ZFFTLCU2FWVLBTKZBSG7UAXZTFSPJUPG6E32E5YH7PRZGTGGIA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.449442888 (592h27m10.320525923s) [noise=true , address=77.90.31.13:4629 , piece_id=N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.560658249 (592h27m10.431741284s) [noise=true , address=77.90.31.16:4735 , piece_id=BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.627821138 (592h27m10.498904183s) [noise=true , address=45.91.116.9:4262 , piece_id=FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.306551407 (592h27m11.177634452s) [noise=true , address=45.91.116.8:4047 , piece_id=GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.522688497 (592h27m11.393771532s) [noise=true , address=145.79.227.43:4157 , piece_id=R5IIHK7IJ57X6R3XRYL4ZF54AKHYAGMF5AMNXDDXHOGXCEX7LLSA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.522709477 (592h27m11.393792512s) [seg_pos={0 0}, upload=519379]: Segment reached optimal threshold of 13 pieces. Dec 30 07:56:11.523023174 (592h27m11.394106219s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523030799 (592h27m11.394113834s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523267459 (592h27m11.394350505s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523274152 (592h27m11.394357187s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523400633 (592h27m11.394483678s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523407145 (592h27m11.39449018s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523423666 (592h27m11.394506711s) [seg_pos={0 0}, upload=519379]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:56:11.523512777 (592h27m11.394595812s) [upload=519379]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:56:11.523518157 (592h27m11.394601192s) [upload=519379]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:56:11.807686789 (592h27m11.678769824s) [upload=519379]: Done uploading segments... Dec 30 07:56:11.807723188 (592h27m11.678806223s) [upload=519379]: Upload finished. err: <nil> Duration: 5.828159812s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X] duration: 5.001669931s trace: https://jaeger.dev.storj.io/trace/2754cbfb05a23cc9 FAIL (2025-12-30 07:57:51 UTC) orbiter warning: context deadline exceeded Duration: 5.001689418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X...X....XXX....X....X.XX..X....XX.XX..................X..] duration: 5.944217714s trace: https://jaeger.dev.storj.io/trace/11cba5b7624f3db7 FAIL (2025-12-30 07:57:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ) context deadline exceeded ======LOGS====== Dec 30 07:56:08.009867651 (592h27m7.880950686s) [upload=519379]: Starting upload Dec 30 07:56:08.009985315 (592h27m7.88106836s) [upload=519379]: Done starting upload Dec 30 07:56:08.010211505 (592h27m7.88129455s) [upload=519379]: Uploading segments... Dec 30 07:56:08.010277080 (592h27m7.881360125s) [upload=519379]: Got next segment. Inline: false Dec 30 07:56:08.010318679 (592h27m7.881401724s) [upload=519379]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:56:08.010323990 (592h27m7.881407035s) [upload=519379]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:56:08.255992922 (592h27m8.127075957s) [seg_pos={0 0}, upload=519379]: Begin upload segment... Dec 30 07:56:08.256053938 (592h27m8.127136973s) [seg_pos={0 0}, upload=519379]: Done begin upload segment. Dec 30 07:56:08.256102701 (592h27m8.127185746s) [noise=true , address=145.79.227.9:4365 , piece_id=SGCMBPUCE5MGMLCKAJOIKZY6VFNRDWI2OTUEHRW4HAI5Q5PGH6JA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256139631 (592h27m8.127222666s) [noise=true , address=145.79.227.43:4157 , piece_id=R5IIHK7IJ57X6R3XRYL4ZF54AKHYAGMF5AMNXDDXHOGXCEX7LLSA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256244531 (592h27m8.127327576s) [noise=true , address=145.79.227.3:4151 , piece_id=MX3WNFBFKBTHIUFURZMDC3JYM4VOZ4QALOYCOVEZAHSMEGFXY4DQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256794608 (592h27m8.127877643s) [noise=true , address=145.79.227.21:4475 , piece_id=KEIPC2WXXWCBYYFYEU6NOP7LV4PMQEN6PFEE57JIK2VRPB2XDVOQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257328394 (592h27m8.128411429s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257403759 (592h27m8.128486804s) [noise=true , address=145.79.227.2:4113 , piece_id=QSERPUV3HAF4VSVMANQGCHBA47JK5PXWNFGSCCFWN55RV5U5DUSQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257613338 (592h27m8.128696383s) [noise=true , address=145.79.227.50:4373 , piece_id=3CX6EMA7L755JW4LFSB3WGTKHM64QXYQ6WSSTMC45JTL5KCDOFBA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257910924 (592h27m8.128993959s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258200745 (592h27m8.129283781s) [noise=true , address=45.91.116.9:4262 , piece_id=FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258510836 (592h27m8.129593881s) [noise=true , address=145.79.227.11:4730 , piece_id=2HYNA6JVHGRQVIS5GOGA2FEQCUJPJVULHXOOLEU6PDR6EIN3O4TQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258883516 (592h27m8.129966551s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259179260 (592h27m8.130262305s) [noise=true , address=145.79.227.52:4445 , piece_id=URXZDZBV3XWVKFYTCC4TRZRBH6M6LZASCF7XUINIIWGPLPEJL2LQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259474712 (592h27m8.130557747s) [noise=true , address=45.91.116.8:4047 , piece_id=GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259848965 (592h27m8.13093201s) [noise=true , address=77.90.31.16:4735 , piece_id=BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.260132514 (592h27m8.131215559s) [noise=true , address=145.79.227.23:4546 , piece_id=27ZFFTLCU2FWVLBTKZBSG7UAXZTFSPJUPG6E32E5YH7PRZGTGGIA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.260712078 (592h27m8.131795113s) [noise=true , address=77.90.31.13:4629 , piece_id=N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:09.952650005 (592h27m9.82373305s) [upload=519379]: Next returned nil segment Dec 30 07:56:09.952662369 (592h27m9.823745404s) [upload=519379]: Waiting for error group managing segments... Dec 30 07:56:10.237249577 (592h27m10.108332643s) [noise=true , address=145.79.227.50:4373 , piece_id=3CX6EMA7L755JW4LFSB3WGTKHM64QXYQ6WSSTMC45JTL5KCDOFBA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.237828901 (592h27m10.108911946s) [noise=true , address=145.79.227.3:4151 , piece_id=MX3WNFBFKBTHIUFURZMDC3JYM4VOZ4QALOYCOVEZAHSMEGFXY4DQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.246031661 (592h27m10.117114706s) [noise=true , address=145.79.227.2:4113 , piece_id=QSERPUV3HAF4VSVMANQGCHBA47JK5PXWNFGSCCFWN55RV5U5DUSQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.250097562 (592h27m10.121180607s) [noise=true , address=145.79.227.21:4475 , piece_id=KEIPC2WXXWCBYYFYEU6NOP7LV4PMQEN6PFEE57JIK2VRPB2XDVOQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.250721180 (592h27m10.121804225s) [noise=true , address=145.79.227.52:4445 , piece_id=URXZDZBV3XWVKFYTCC4TRZRBH6M6LZASCF7XUINIIWGPLPEJL2LQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.276428340 (592h27m10.147511385s) [noise=true , address=145.79.227.11:4730 , piece_id=2HYNA6JVHGRQVIS5GOGA2FEQCUJPJVULHXOOLEU6PDR6EIN3O4TQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.285513861 (592h27m10.156596906s) [noise=true , address=145.79.227.9:4365 , piece_id=SGCMBPUCE5MGMLCKAJOIKZY6VFNRDWI2OTUEHRW4HAI5Q5PGH6JA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.333682441 (592h27m10.204765486s) [noise=true , address=145.79.227.23:4546 , piece_id=27ZFFTLCU2FWVLBTKZBSG7UAXZTFSPJUPG6E32E5YH7PRZGTGGIA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.449442888 (592h27m10.320525923s) [noise=true , address=77.90.31.13:4629 , piece_id=N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.560658249 (592h27m10.431741284s) [noise=true , address=77.90.31.16:4735 , piece_id=BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.627821138 (592h27m10.498904183s) [noise=true , address=45.91.116.9:4262 , piece_id=FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.306551407 (592h27m11.177634452s) [noise=true , address=45.91.116.8:4047 , piece_id=GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.522688497 (592h27m11.393771532s) [noise=true , address=145.79.227.43:4157 , piece_id=R5IIHK7IJ57X6R3XRYL4ZF54AKHYAGMF5AMNXDDXHOGXCEX7LLSA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.522709477 (592h27m11.393792512s) [seg_pos={0 0}, upload=519379]: Segment reached optimal threshold of 13 pieces. Dec 30 07:56:11.523023174 (592h27m11.394106219s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523030799 (592h27m11.394113834s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523267459 (592h27m11.394350505s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523274152 (592h27m11.394357187s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523400633 (592h27m11.394483678s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523407145 (592h27m11.39449018s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523423666 (592h27m11.394506711s) [seg_pos={0 0}, upload=519379]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:56:11.523512777 (592h27m11.394595812s) [upload=519379]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:56:11.523518157 (592h27m11.394601192s) [upload=519379]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:56:11.807686789 (592h27m11.678769824s) [upload=519379]: Done uploading segments... Dec 30 07:56:11.807723188 (592h27m11.678806223s) [upload=519379]: Upload finished. err: <nil> Duration: 5.944247031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 27.319763332s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:57:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 27.319790714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X...X....XXX....X....X.XX..X....XX.XX..................X.] duration: 6.066608743s trace: https://jaeger.dev.storj.io/trace/11cba5b7624f3db7 FAIL (2025-12-30 07:57:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ) context deadline exceeded ======LOGS====== Dec 30 07:56:08.009867651 (592h27m7.880950686s) [upload=519379]: Starting upload Dec 30 07:56:08.009985315 (592h27m7.88106836s) [upload=519379]: Done starting upload Dec 30 07:56:08.010211505 (592h27m7.88129455s) [upload=519379]: Uploading segments... Dec 30 07:56:08.010277080 (592h27m7.881360125s) [upload=519379]: Got next segment. Inline: false Dec 30 07:56:08.010318679 (592h27m7.881401724s) [upload=519379]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:56:08.010323990 (592h27m7.881407035s) [upload=519379]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:56:08.255992922 (592h27m8.127075957s) [seg_pos={0 0}, upload=519379]: Begin upload segment... Dec 30 07:56:08.256053938 (592h27m8.127136973s) [seg_pos={0 0}, upload=519379]: Done begin upload segment. Dec 30 07:56:08.256102701 (592h27m8.127185746s) [noise=true , address=145.79.227.9:4365 , piece_id=SGCMBPUCE5MGMLCKAJOIKZY6VFNRDWI2OTUEHRW4HAI5Q5PGH6JA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256139631 (592h27m8.127222666s) [noise=true , address=145.79.227.43:4157 , piece_id=R5IIHK7IJ57X6R3XRYL4ZF54AKHYAGMF5AMNXDDXHOGXCEX7LLSA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256244531 (592h27m8.127327576s) [noise=true , address=145.79.227.3:4151 , piece_id=MX3WNFBFKBTHIUFURZMDC3JYM4VOZ4QALOYCOVEZAHSMEGFXY4DQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256794608 (592h27m8.127877643s) [noise=true , address=145.79.227.21:4475 , piece_id=KEIPC2WXXWCBYYFYEU6NOP7LV4PMQEN6PFEE57JIK2VRPB2XDVOQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257328394 (592h27m8.128411429s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257403759 (592h27m8.128486804s) [noise=true , address=145.79.227.2:4113 , piece_id=QSERPUV3HAF4VSVMANQGCHBA47JK5PXWNFGSCCFWN55RV5U5DUSQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257613338 (592h27m8.128696383s) [noise=true , address=145.79.227.50:4373 , piece_id=3CX6EMA7L755JW4LFSB3WGTKHM64QXYQ6WSSTMC45JTL5KCDOFBA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257910924 (592h27m8.128993959s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258200745 (592h27m8.129283781s) [noise=true , address=45.91.116.9:4262 , piece_id=FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258510836 (592h27m8.129593881s) [noise=true , address=145.79.227.11:4730 , piece_id=2HYNA6JVHGRQVIS5GOGA2FEQCUJPJVULHXOOLEU6PDR6EIN3O4TQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258883516 (592h27m8.129966551s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259179260 (592h27m8.130262305s) [noise=true , address=145.79.227.52:4445 , piece_id=URXZDZBV3XWVKFYTCC4TRZRBH6M6LZASCF7XUINIIWGPLPEJL2LQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259474712 (592h27m8.130557747s) [noise=true , address=45.91.116.8:4047 , piece_id=GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259848965 (592h27m8.13093201s) [noise=true , address=77.90.31.16:4735 , piece_id=BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.260132514 (592h27m8.131215559s) [noise=true , address=145.79.227.23:4546 , piece_id=27ZFFTLCU2FWVLBTKZBSG7UAXZTFSPJUPG6E32E5YH7PRZGTGGIA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.260712078 (592h27m8.131795113s) [noise=true , address=77.90.31.13:4629 , piece_id=N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:09.952650005 (592h27m9.82373305s) [upload=519379]: Next returned nil segment Dec 30 07:56:09.952662369 (592h27m9.823745404s) [upload=519379]: Waiting for error group managing segments... Dec 30 07:56:10.237249577 (592h27m10.108332643s) [noise=true , address=145.79.227.50:4373 , piece_id=3CX6EMA7L755JW4LFSB3WGTKHM64QXYQ6WSSTMC45JTL5KCDOFBA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.237828901 (592h27m10.108911946s) [noise=true , address=145.79.227.3:4151 , piece_id=MX3WNFBFKBTHIUFURZMDC3JYM4VOZ4QALOYCOVEZAHSMEGFXY4DQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.246031661 (592h27m10.117114706s) [noise=true , address=145.79.227.2:4113 , piece_id=QSERPUV3HAF4VSVMANQGCHBA47JK5PXWNFGSCCFWN55RV5U5DUSQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.250097562 (592h27m10.121180607s) [noise=true , address=145.79.227.21:4475 , piece_id=KEIPC2WXXWCBYYFYEU6NOP7LV4PMQEN6PFEE57JIK2VRPB2XDVOQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.250721180 (592h27m10.121804225s) [noise=true , address=145.79.227.52:4445 , piece_id=URXZDZBV3XWVKFYTCC4TRZRBH6M6LZASCF7XUINIIWGPLPEJL2LQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.276428340 (592h27m10.147511385s) [noise=true , address=145.79.227.11:4730 , piece_id=2HYNA6JVHGRQVIS5GOGA2FEQCUJPJVULHXOOLEU6PDR6EIN3O4TQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.285513861 (592h27m10.156596906s) [noise=true , address=145.79.227.9:4365 , piece_id=SGCMBPUCE5MGMLCKAJOIKZY6VFNRDWI2OTUEHRW4HAI5Q5PGH6JA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.333682441 (592h27m10.204765486s) [noise=true , address=145.79.227.23:4546 , piece_id=27ZFFTLCU2FWVLBTKZBSG7UAXZTFSPJUPG6E32E5YH7PRZGTGGIA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.449442888 (592h27m10.320525923s) [noise=true , address=77.90.31.13:4629 , piece_id=N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.560658249 (592h27m10.431741284s) [noise=true , address=77.90.31.16:4735 , piece_id=BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.627821138 (592h27m10.498904183s) [noise=true , address=45.91.116.9:4262 , piece_id=FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.306551407 (592h27m11.177634452s) [noise=true , address=45.91.116.8:4047 , piece_id=GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.522688497 (592h27m11.393771532s) [noise=true , address=145.79.227.43:4157 , piece_id=R5IIHK7IJ57X6R3XRYL4ZF54AKHYAGMF5AMNXDDXHOGXCEX7LLSA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.522709477 (592h27m11.393792512s) [seg_pos={0 0}, upload=519379]: Segment reached optimal threshold of 13 pieces. Dec 30 07:56:11.523023174 (592h27m11.394106219s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523030799 (592h27m11.394113834s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523267459 (592h27m11.394350505s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523274152 (592h27m11.394357187s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523400633 (592h27m11.394483678s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523407145 (592h27m11.39449018s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523423666 (592h27m11.394506711s) [seg_pos={0 0}, upload=519379]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:56:11.523512777 (592h27m11.394595812s) [upload=519379]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:56:11.523518157 (592h27m11.394601192s) [upload=519379]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:56:11.807686789 (592h27m11.678769824s) [upload=519379]: Done uploading segments... Dec 30 07:56:11.807723188 (592h27m11.678806223s) [upload=519379]: Upload finished. err: <nil> Duration: 6.066636067s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......] duration: 3.383297835s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:57:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383324646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 31.387399441s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:56:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 31.387429829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..X...X....XXX....X....X.XX..X....XX.XX..................X] duration: 30.002970565s trace: https://jaeger.dev.storj.io/trace/11cba5b7624f3db7 FAIL (2025-12-30 07:56:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ) context deadline exceeded ======LOGS====== Dec 30 07:56:08.009867651 (592h27m7.880950686s) [upload=519379]: Starting upload Dec 30 07:56:08.009985315 (592h27m7.88106836s) [upload=519379]: Done starting upload Dec 30 07:56:08.010211505 (592h27m7.88129455s) [upload=519379]: Uploading segments... Dec 30 07:56:08.010277080 (592h27m7.881360125s) [upload=519379]: Got next segment. Inline: false Dec 30 07:56:08.010318679 (592h27m7.881401724s) [upload=519379]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:56:08.010323990 (592h27m7.881407035s) [upload=519379]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:56:08.255992922 (592h27m8.127075957s) [seg_pos={0 0}, upload=519379]: Begin upload segment... Dec 30 07:56:08.256053938 (592h27m8.127136973s) [seg_pos={0 0}, upload=519379]: Done begin upload segment. Dec 30 07:56:08.256102701 (592h27m8.127185746s) [noise=true , address=145.79.227.9:4365 , piece_id=SGCMBPUCE5MGMLCKAJOIKZY6VFNRDWI2OTUEHRW4HAI5Q5PGH6JA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256139631 (592h27m8.127222666s) [noise=true , address=145.79.227.43:4157 , piece_id=R5IIHK7IJ57X6R3XRYL4ZF54AKHYAGMF5AMNXDDXHOGXCEX7LLSA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256244531 (592h27m8.127327576s) [noise=true , address=145.79.227.3:4151 , piece_id=MX3WNFBFKBTHIUFURZMDC3JYM4VOZ4QALOYCOVEZAHSMEGFXY4DQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.256794608 (592h27m8.127877643s) [noise=true , address=145.79.227.21:4475 , piece_id=KEIPC2WXXWCBYYFYEU6NOP7LV4PMQEN6PFEE57JIK2VRPB2XDVOQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257328394 (592h27m8.128411429s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257403759 (592h27m8.128486804s) [noise=true , address=145.79.227.2:4113 , piece_id=QSERPUV3HAF4VSVMANQGCHBA47JK5PXWNFGSCCFWN55RV5U5DUSQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257613338 (592h27m8.128696383s) [noise=true , address=145.79.227.50:4373 , piece_id=3CX6EMA7L755JW4LFSB3WGTKHM64QXYQ6WSSTMC45JTL5KCDOFBA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.257910924 (592h27m8.128993959s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258200745 (592h27m8.129283781s) [noise=true , address=45.91.116.9:4262 , piece_id=FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258510836 (592h27m8.129593881s) [noise=true , address=145.79.227.11:4730 , piece_id=2HYNA6JVHGRQVIS5GOGA2FEQCUJPJVULHXOOLEU6PDR6EIN3O4TQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.258883516 (592h27m8.129966551s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259179260 (592h27m8.130262305s) [noise=true , address=145.79.227.52:4445 , piece_id=URXZDZBV3XWVKFYTCC4TRZRBH6M6LZASCF7XUINIIWGPLPEJL2LQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259474712 (592h27m8.130557747s) [noise=true , address=45.91.116.8:4047 , piece_id=GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.259848965 (592h27m8.13093201s) [noise=true , address=77.90.31.16:4735 , piece_id=BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.260132514 (592h27m8.131215559s) [noise=true , address=145.79.227.23:4546 , piece_id=27ZFFTLCU2FWVLBTKZBSG7UAXZTFSPJUPG6E32E5YH7PRZGTGGIA, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:08.260712078 (592h27m8.131795113s) [noise=true , address=77.90.31.13:4629 , piece_id=N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ, seg_pos={0 0}, upload=519379]: Uploading piece... Dec 30 07:56:09.952650005 (592h27m9.82373305s) [upload=519379]: Next returned nil segment Dec 30 07:56:09.952662369 (592h27m9.823745404s) [upload=519379]: Waiting for error group managing segments... Dec 30 07:56:10.237249577 (592h27m10.108332643s) [noise=true , address=145.79.227.50:4373 , piece_id=3CX6EMA7L755JW4LFSB3WGTKHM64QXYQ6WSSTMC45JTL5KCDOFBA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.237828901 (592h27m10.108911946s) [noise=true , address=145.79.227.3:4151 , piece_id=MX3WNFBFKBTHIUFURZMDC3JYM4VOZ4QALOYCOVEZAHSMEGFXY4DQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.246031661 (592h27m10.117114706s) [noise=true , address=145.79.227.2:4113 , piece_id=QSERPUV3HAF4VSVMANQGCHBA47JK5PXWNFGSCCFWN55RV5U5DUSQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.250097562 (592h27m10.121180607s) [noise=true , address=145.79.227.21:4475 , piece_id=KEIPC2WXXWCBYYFYEU6NOP7LV4PMQEN6PFEE57JIK2VRPB2XDVOQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.250721180 (592h27m10.121804225s) [noise=true , address=145.79.227.52:4445 , piece_id=URXZDZBV3XWVKFYTCC4TRZRBH6M6LZASCF7XUINIIWGPLPEJL2LQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.276428340 (592h27m10.147511385s) [noise=true , address=145.79.227.11:4730 , piece_id=2HYNA6JVHGRQVIS5GOGA2FEQCUJPJVULHXOOLEU6PDR6EIN3O4TQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.285513861 (592h27m10.156596906s) [noise=true , address=145.79.227.9:4365 , piece_id=SGCMBPUCE5MGMLCKAJOIKZY6VFNRDWI2OTUEHRW4HAI5Q5PGH6JA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.333682441 (592h27m10.204765486s) [noise=true , address=145.79.227.23:4546 , piece_id=27ZFFTLCU2FWVLBTKZBSG7UAXZTFSPJUPG6E32E5YH7PRZGTGGIA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.449442888 (592h27m10.320525923s) [noise=true , address=77.90.31.13:4629 , piece_id=N7NID2DDENCYPSEFXHWACENE3FGONFODAXGGAFHC5G2THB6AC7YQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.560658249 (592h27m10.431741284s) [noise=true , address=77.90.31.16:4735 , piece_id=BWJI4TUBMXYJL6HMHN2OSB4N3DCOBJDFYCBCHJG4D4UA542VHRRA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:10.627821138 (592h27m10.498904183s) [noise=true , address=45.91.116.9:4262 , piece_id=FVEZU2I7C2IZBHZG7YDCALSROTH2T3Q7K2BC747YN47JV4NEVWYQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.306551407 (592h27m11.177634452s) [noise=true , address=45.91.116.8:4047 , piece_id=GDQBOYWQWSNQSRMB26UCAUX4X27KNRWG2XG7PZVCHI4RLEXVE5NA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.522688497 (592h27m11.393771532s) [noise=true , address=145.79.227.43:4157 , piece_id=R5IIHK7IJ57X6R3XRYL4ZF54AKHYAGMF5AMNXDDXHOGXCEX7LLSA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: <nil> Dec 30 07:56:11.522709477 (592h27m11.393792512s) [seg_pos={0 0}, upload=519379]: Segment reached optimal threshold of 13 pieces. Dec 30 07:56:11.523023174 (592h27m11.394106219s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523030799 (592h27m11.394113834s) [noise=true , address=77.90.31.21:4877 , piece_id=KIF7NF5VCLJX2LPYCQEOB4CUUGMZBXVTD4534DREALZVMFLW6TRQ, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523267459 (592h27m11.394350505s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523274152 (592h27m11.394357187s) [noise=true , address=77.90.31.11:4554 , piece_id=GW2HJ5JRSDZGKWGJ5PVZDU4WFYETXWYSKWHSCHWYJP7QYR5BUGXQ, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523400633 (592h27m11.394483678s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:56:11.523407145 (592h27m11.39449018s) [noise=true , address=45.91.116.4:4939 , piece_id=SSDCGPV4F2VE5FTXLJYGMDAOEIYNR7NIPCESYHVBEWJJKBEUHQWA, seg_pos={0 0}, upload=519379]: Long tail cancellation due to optimal threshold Dec 30 07:56:11.523423666 (592h27m11.394506711s) [seg_pos={0 0}, upload=519379]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:56:11.523512777 (592h27m11.394595812s) [upload=519379]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:56:11.523518157 (592h27m11.394601192s) [upload=519379]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:56:11.807686789 (592h27m11.678769824s) [upload=519379]: Done uploading segments... Dec 30 07:56:11.807723188 (592h27m11.678806223s) [upload=519379]: Upload finished. err: <nil> Duration: 30.002995071s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX......] duration: 3.552612705s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:56:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552644617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 44.688844463s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:56:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 44.688871825s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.....] duration: 3.476652948s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:55:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476684377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003119407s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:55:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 1m0.003130839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX..X...X....XXX....X....X.XX..X....XX.XX..................] duration: 6.138420904s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:55:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 6.13844499s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX....] duration: 3.561667517s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:54:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.561696282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX...] duration: 4.054797835s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:54:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.054873211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.XX..X...X....XXX....X....X.XX..X....XX.XX.................] duration: 6.342812448s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:54:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 6.342856351s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX..] duration: 3.589985069s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:53:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.590008964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 23.739530623s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:53:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 23.739562675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.] duration: 3.520884109s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:53:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520917162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.XX..X...X....XXX....X....X.XX..X....XX.XX................] duration: 5.951798776s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:53:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.951828091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 33.637760619s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 33.637788522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.XX..X...X....XXX....X....X.XX..X....XX.XX...............] duration: 6.148225675s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 6.148252475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X.XX..X...X....XXX....X....X.XX..X....XX.XX..............] duration: 25.708510043s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:52:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 25.708539448s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX] duration: 5.00111886s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:51:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001136714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 18.007248103s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:51:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 18.00726768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.XX..X...X....XXX....X....X.XX..X....XX.XX.............] duration: 5.667796771s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:51:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.667828371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 44.000746158s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:51:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 44.000771436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.X] duration: 5.000962772s trace: https://jaeger.dev.storj.io/trace/6f554937c8a51d2c FAIL (2025-12-30 07:51:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0010666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.] duration: 3.615956474s trace: https://jaeger.dev.storj.io/trace/7e75adec67fafb7f FAIL (2025-12-30 07:50:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.615982894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.XX..X...X....XXX....X....X.XX..X....XX.XX............] duration: 5.668237702s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:50:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.668273118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 21.91329559s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:50:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 21.913312573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [...X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX] duration: 5.001665622s trace: https://jaeger.dev.storj.io/trace/7e75adec67fafb7f FAIL (2025-12-30 07:49:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001683906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 50.72595334s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:49:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 50.725976453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXX] duration: 5.002001361s trace: https://jaeger.dev.storj.io/trace/338caacdfafc8965 FAIL (2025-12-30 07:49:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002013584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX...........] duration: 7.404994698s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:49:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 7.405027401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 50.422096433s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:48:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 50.422125748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX..........] duration: 5.629074937s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:48:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.629101929s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXX] duration: 5.001973681s trace: https://jaeger.dev.storj.io/trace/69f3438ddc840f09 FAIL (2025-12-30 07:48:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002000581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX.........] duration: 5.633262082s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.633291077s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)